Change the strings

To expand puppet farther, we should look at some basics and get a better understanding of things.
Out of the box puppet lets you do some fun things, but its reporting and monitoring is a bit lacking unless you buy Puppet Enterprise
So yes there are other ways to get reporting, these are just the ways I found in 2015 to get some insite into things and it starts here.

Start with I am on Ubuntu systems, so that might be a factor, but just replace the apt-get with yummy install.

Let's start by making the basic assumption that your puppet server is not a shared resource server, or this will end badly. Ok not this step but any future steps would.

Let's do the simple install of them from the pre-existing repo you used to install puppetmaster.

root@puppet02:~# sudo apt-get install puppetdb puppetdb-terminus 

The first thing you want to do after the install is change the heap size. This is in /etc/default/puppetdb

# Modify this if you'd like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

You want to make the -Xmx192m to at least 1GB for under 100 servers

# Modify this if you'd like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -Djava.security.egd=file:/dev/urandom"

Next is tying it to puppetmaster so create the file /etc/puppet/puppetdb.conf  and add

[main]
server = puppetdb.example.com
port = 8081
soft_write_failure = false

Next you want puppet to be able to know where to put the confis and the reports so lets tell it in the /etc/puppet/puppet.conf
I add these lines under the [master] section

storeconfigs = true
storeconfigs_backen = puppetdb
reports = store,puppetdb

Next we add the routes so it knows where to look, the default location is /etc/puppet/routes.yaml so im putting it there

---
master:
  facts:
    terminus: puppetdb
    cache: yaml

In the offical instructions it says to verify permissions, so lets do that

root@puppet02:~# sudo chown -R puppet:puppet /etc/puppet 

 Next step is to allow the connections to the /etc/puppetdb/conf.d/jetty.ini
My config is below, I have only added the line host = 0.0.0.0

[jetty]
# IP address or hostname to listen for clear-text HTTP. To avoid resolution
# issues, IP addresses are recommended over hostnames.
# Default is `localhost`.
# host = <host>
host = 0.0.0.0
# Port to listen on for clear-text HTTP.
port = 8080


# The following are SSL specific settings. They can be configured
# automatically with the tool `puppetdb ssl-setup`, which is normally
# ran during package installation.

# IP address to listen on for HTTPS connections. Hostnames can also be used
# but are not recommended to avoid DNS resolution issues. To listen on all
# interfaces, use `0.0.0.0`.
ssl-host = 0.0.0.0

# The port to listen on for HTTPS connections
ssl-port = 8081

# Private key path
ssl-key = /etc/puppetdb/ssl/private.pem

# Public certificate path
ssl-cert = /etc/puppetdb/ssl/public.pem

# Certificate authority path
ssl-ca-cert = /etc/puppetdb/ssl/ca.pem

Now we want to make sure the certificates are valid. So lets do a puppetdb ssl-setup

root@puppet02:~# puppetdb ssl-setup -f 
PEM files in /etc/puppetdb/ssl already exists, checking integrity.
Overwriting existing PEM files due to -f flag
Copying files: /var/lib/puppet/ssl/certs/ca.pem, /var/lib/puppet/ssl/private_keys/puppet2.velcrohurts.com.pem and /var/lib/puppet/ssl/certs/puppet2.velcrohurts.com.pem to /etc/puppetdb/ssl
Setting ssl-host in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-port in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-key in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-cert in /etc/puppetdb/conf.d/jetty.ini already correct.
Setting ssl-ca-cert in /etc/puppetdb/conf.d/jetty.ini already correct.

Then we should be able to restart the puppetmaster service and the puppetdb service
We know it works when the /var/log/puppetdb/puppetdb.log is tailed and it shows the connections
and if everything works right, we should have the output of

2015-07-17 15:21:58,104 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@788953f5{/,null,AVAILABLE}
2015-07-17 15:21:58,120 INFO  [c.p.p.c.services] Starting sweep of stale reports (threshold: 14 days)
2015-07-17 15:21:58,161 INFO  [c.p.p.c.services] Finished sweep of stale reports (threshold: 14 days)
2015-07-17 15:21:58,162 INFO  [c.p.p.c.services] Starting database garbage collection
2015-07-17 15:21:58,250 INFO  [c.p.p.c.services] Finished database garbage collection

Variables May Vary

Now that we have talked about puppet some, let's add some variables to the mix.
The simple idea is that they are passed from the puppet/manifest/site.pp to the module/apache/templates/velcrohurts.conf.erb via magic!

I agree, it is magic, but the magic tells you how to do it.
Lets start with breaking down our previous simple config.

# Managed by Puppet 
# All changes will be overwriten 

<VirtualHost *:80>
  ServerName internal.velcrohurts.net 
  DocumentRoot  /var/www/html/
  LogLevel info 
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" combined
  ErrorLog ${APACHE_LOG_DIR}/velcrohurts-error.log
  CustomLog ${APACHE_LOG_DIR}/velcrohurts-access.log combined
  RedirectMatch ^/$ http://velcrohurts.net/
  <Directory /var/www/html/> 
     AllowOverride all
     Order allow,deny 
     allow from all
  </Directory>
</virtualHost> 

The virtual host config has the ServerName, what if we want to use this config for other servers.
We can edit it, or we can set it as a variable. Let's do the second one for this example.

First we need to copy the config to the modules/apache/templates/velcrohurts.conf.erb

Then we want to edit it to add the variable names.

# Managed by Puppet 
# All changes will be overwriten 

<VirtualHost *:80>
  ServerName <%= server_name %>  
  DocumentRoot  /var/www/html/
  LogLevel info 
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" combined
  ErrorLog ${APACHE_LOG_DIR}/<%= server_name %>-error.log

CustomLog ${APACHE_LOG_DIR}/<%= server_name %>-access.log combined
  RedirectMatch ^/$ http://velcrohurts.net/
  <Directory /var/www/html/> 
     AllowOverride all
     Order allow,deny 
     allow from all
  </Directory>
</virtualHost> 

So now that we have the .erb we need to edit our modules/apache/manifests/init.pp and add the variables there, and make a few changes.

class apache ($server_name){ 
# Install package 
    package {'apache2':
        ensure  => installed,
    }

# set the configure for debian 
  file { '/etc/apache2/sites-available/velcrohurts.conf':
    ensure => file,
    mode   => '0644',
    owner  => 'root',
    group  => 'root',
    content => template["apache/velcrohurts.conf.erb"],
    require => Package['apache2'],
    notify => Exec ["reload-apache2"],
  }

# Reload apache with the config 
# And remove the default-000 config 
    exec {'/usr/sbin/a2dissite 000-default':
    require => Package['apache2'],
    notify => Exec ["reload-apache2"],
    }

# Now add your config 
    exec {'/usr/sbin/a2ensite ajplus':
      notify => Exec ["reload-apache2"],
      require => Package['apache2'],
    }

# the apache reloader 
    exec { 'reload-apache2':
      command => '/etc/init.d/apache2 reload',
      refreshonly => true,
  }
}

The final step here is to add the magic, set the manifests/site.pp settings

node default { 
   include accounts 
} 

# web servers 
node web1 { 
   class apache {
       server_name = internal.velcrohurts.com
} 
node web2 { 
   class apache {
       server_name = public.velcrohurts.com
} 
node web3 { 
   class apache {
       server_name = public.velcrohurts.com
} 

Do it again

How about a super simple apache install?
This is just a really simple installation of apache, and it can be done with any application. I might even have a short cut in my text expander.

this is the modules/apache/manifests/init.pp

class apache { 
# Install package 
    package {'apache2':
        ensure  => installed,
    }

# set the configure for debian 
  file { '/etc/apache2/sites-available/velcrohurts.conf':
    ensure => file,
    mode   => '0644',
    owner  => 'root',
    group  => 'root',
    source => 'puppet:///modules/apache/velcrohurts.conf',
    require => Package['apache2'],
    notify => Exec ["reload-apache2"],
  }

# Reload apache with the config 
# And remove the default-000 config 
    exec {'/usr/sbin/a2dissite 000-default':
    require => Package['apache2'],
    notify => Exec ["reload-apache2"],
    }

# Now add your config 
    exec {'/usr/sbin/a2ensite ajplus':
      notify => Exec ["reload-apache2"],
      require => Package['apache2'],
    }

# the apache reloader 
    exec { 'reload-apache2':
      command => '/etc/init.d/apache2 reload',
      refreshonly => true,
  }
}

That just keeps it nice and simple. It installs the apache app, sets the config, and restarts apache.
My config is simple because I only have 1 site per server, so my modules/apache/files/velcrohurts.conf looks a lot like this.

# Managed by Puppet 
# All changes will be overwriten 

<VirtualHost *:80>
  ServerName internal.velcrohurts.net 
  DocumentRoot  /var/www/html/
  LogLevel info 
  LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b" combined
  ErrorLog ${APACHE_LOG_DIR}/velcrohurts-error.log
  CustomLog ${APACHE_LOG_DIR}/velcrohurts-access.log combined
  RedirectMatch ^/$ http://velcrohurts.net/
  <Directory /var/www/html/> 
     AllowOverride all
     Order allow,deny 
     allow from all
  </Directory>
</virtualHost> 

Then finally, your module should look like this under your manifests/site.pp

node default { 
   include accounts 
} 

# web servers 
node web1 { 
    include apache 
} 

node web2 { 
    include apache 
} 

node web3 { 
    include apache 
} 

There are no strings on me!

Lets start with puppet.
I think I made a mistake and made it too complicated out of the gate. I did some bad things in my configs, like set environments for different things.
So lets review a basic puppetmaster config with environments.
It is in the standard /etc/puppet/puppet.conf location

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
environmentpath = $confdir/environments
basemodulepath = $confdir/modules
dns_alt_names = puppet,puppet.velcrohurts.com,puppet.velcrohurts.local
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

server = puppet.velcrohurts.com
runinterval = 300

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN 
ssl_client_verify_header = SSL_CLIENT_VERIFY

So that little bit tells us we want to use environments in /etc/puppet/environments/
inside that directory we have

stephen.mcgroarty@nostrings:~$ ls /etc/puppet/environments/
development  example_env  production  staging

I think the labels are self explanatory here, but you can add a folder for whatever environment you want.
I recommend usign the environment =  tag on the clients.
I actually have a puppet manifest for the client side, so that when I put them in it stays there.

For the client, I have this manifest under /etc/puppet/environments/staging/modules/puppet/manifests/init.pp

class puppet {

    file { '/etc/puppet/puppet.conf':
        ensure => file,
        mode   => '644',
        owner  => 'root',
        group  => 'root',
        source => 'puppet:///modules/puppet/puppet.conf.erb'
    }
}

Then for the modules/puppet/files/puppet.conf.erb I have the puppet config.

## Managed by Puppet ## 

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter


server = puppet.velcrohurts.local
environment = staging
runinterval = 500

I change the environment = based on what location I want, and using puppet to control the puppet.conf might seem a bit odd, but it never changes on a single server for very long.

Quick and Dirty

Let  me make this a quick and dirty one because I just want to post something slightly useful. So no re-write, no review, the errors stay for the most part.

Here is a git command line shortcut that I use almost daily, and it has been really easy to use from the command prompt, it is an alias for lol

[alias]
        lol = log --graph --decorate --pretty=oneline --abbrev-commit
        lola = log --graph --decorate --pretty=oneline --abbrev-commit --all

Once you have it in your /home/username/.gitconfig you can do the cool thing of laughing out loud at git, and some people do, but I dont enjoy typing lol

The output looks all nice and happy

git lol

vs 

git log 

The choice is yours, but I like pretty colors and shorter blocks. 

Works for me!

One of the more interesting things I have had to do lately was setup a site to site VPN in AWS using openswan.
After reading all the notes, blog posts, forum posts I was able to luck onto a configuration that works for me, your milage will vary.

Spin up your new instances in AWS, details can be found elsewhere, but once you get the new instances spun up in each location you can start the configuration parts.

After the instance is deployed right click on it and select Networking > Change Source/Dest. Check and then select Enable

If you do not enable this setting, you will not get full two way traffic, so be warned.


Now we want to add the routing to the VPC the select an Elastic IP for the instances and assign it. Once per site. Write down the Network Interface ID, eni-XXXXXXXX

While under the VPC select Route Tables and then your VPC
Edit > Add Another Route
The next part put your remote subnet and the Network Interface ID, eni-XXXXXXXX and save that.

For the security parts, you need these.
If you had your instance in the default group, that is fine, but here we want to change it.
I usually make my own firewall rules for this part, use your own judgement here.

You need UDP 500 and UDP 4500.
You also need ESP 50

I usually do All Traffic to the remote public IP
And then I do Custom Protocol type in 50, and then all ports to the remote IP

The sg- sources are the other security groups, you need to to have those for all the subnets to talk to each other.

Next do you apt-get/yum install openswan, it is part of the standard ubuntu packages. You also want to install ipsec-tools on Ubuntu.

Now we get to the "works for me" part of this. After reading the openswan manual, may forum, blog and news group posts, I discovered the that I needed more settings then they were showing. So here we go.

conn conn-name-east
 type=tunnel 
 left=10.1.1.101 # your private IP of this server 
 leftsubnets=10.1.0.0/16 # Full VPC subnet
 leftid=52.XX.XX.XX #Your public IP of this server 
 leftsourceip=10.1.1.101
 right=54.XX.XX.XX   # The remote server IP 
 rightsubnets=10.2.0.0/16
 rightid=54.XX.XX.XX  #The other IP
 pfs=no
 forceencaps=yes
 authby=secret
 auto=start

Now that is a great start, just change the values accordingly for your other connection

conn conn-name-west
 type=tunnel 
 left=10.2.1.101 # your private IP of this server 
 leftsubnets=10.2.0.0/16 # Full VPC subnet
 leftid=54.XX.XX.XX #Your public IP of this server 
 leftsourceip=10.2.1.101
 right=52.XX.XX.XX   # The remote server IP 
 rightsubnets=10.1.0.0/16
 rightid=52.XX.XX.XX  #The other IP
 pfs=no
 forceencaps=yes
 authby=secret
 auto=start

Shhh.. it's a secret..
So above we are telling it to auth by secret, I am not sure this is the most secure way to do things, but it does seem to work so far.
On each side I am using the public IP and remote IP in the secrets.

54.XX.XX.XX 52.XX.XX.XX: PSK "$VER_SEC_PSK"

Now for the commands that need to be run on both servers.
We need to tell our new happy servers that, yes they can talk, and yes they can forward information to the networks
In the /etc/sysctl.conf you want to clear out what is there and add these values.

net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.eth0.rp_filter=0
net.ipv4.conf.lo.rp_filter=0
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.default.send_redirects=0
net.ipv4.conf.eth0.send_redirects=0
net.ipv4.conf.lo.send_redirects=0
net.ipv4.conf.all.accept_redirects=0
net.ipv4.conf.default.accept_redirects=0
net.ipv4.conf.eth0.accept_redirects=0
net.ipv4.conf.lo.accept_redirects=0
net.ipv4.ip_forward=1

If you do not want to restart the system, type paste these commands so the settings take effect instantly.

echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects 
echo 0 > /proc/sys/net/ipv4/conf/default/accept_redirects 
echo 0 > /proc/sys/net/ipv4/conf/eth0/accept_redirects 
echo 0 > /proc/sys/net/ipv4/conf/lo/accept_redirects 
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/default/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/eth0/send_redirects
echo 0 > /proc/sys/net/ipv4/conf/lo/send_redirects

If you are not able to ping, run this command on the machine, go ahead try it, it is fun.

iptables -t nat  -A POSTROUTING -s localsubnet/16 ! -d remotesubnet/16 -o eth0 -j MASQUERADE

That last command took me a while, because if you notice I am relying on AWS firewalls to protect me, not the system itself. I want the system to be able to communicate with everything else, on a select IP.

For simple verification you can run the sudo service ipsec status  command

ubuntu@myawesomevpnserver-east:~$ sudo service ipsec status
IPsec running  - pluto pid: 2282
pluto pid 2282
1 tunnels up
some eroutes exist

Some useful commands and their outputs.
ipsec verify Used to make sure your configurations are in tact

ubuntu@myawesomevpnserver-east:~$  sudo ipsec verify 
Checking your system to see if IPsec got installed and started correctly:
Version check and ipsec on-path                              [OK]
Linux Openswan U2.6.38/K3.13.0-44-generic (netkey)
Checking for IPsec support in kernel                         [OK]
 SAref kernel support                                        [N/A]
 NETKEY:  Testing XFRM related proc values                   [OK]
 [OK]
 [OK]
Checking that pluto is running                               [OK]
 Pluto listening for IKE on udp 500                          [OK]
 Pluto listening for NAT-T on udp 4500                       [OK]
Checking for 'ip' command                                    [OK]
Checking /bin/sh is not /bin/dash                            [WARNING]
Checking for 'iptables' command                              [OK]
Opportunistic Encryption Support                             [DISABLED]

ipsec auto status  make sure you are connected and the tunnel is up. you mainly care about the last 4 lines here, there is a lot of output

ubuntu@myawesomevpnserver-east:~$  sudo ipsec auto status 
ipsec auto: warning: obsolete command syntax used
000 using kernel interface: netkey
000 interface lo/lo ::1
000 interface lo/lo 127.0.0.1
000 interface lo/lo 127.0.0.1
---- SNIP ---- 
000 #939: "conn-name-east/1x1":4500 STATE_MAIN_R3 (sent MR3, ISAKMP SA established); EVENT_SA_REPLACE in 3269s; newest ISAKMP; lastdpd=-1s(seq in:0 out:0); idle; import:not set
000 #938: "conn-name-east/1x1":4500 STATE_MAIN_R3 (sent MR3, ISAKMP SA established); EVENT_SA_REPLACE in 672s; lastdpd=-1s(seq in:0 out:0); idle; import:not set
000 #930: "conn-name-east/1x1":4500 STATE_QUICK_R2 (IPsec SA established); EVENT_SA_REPLACE in 4875s; newest IPSEC; eroute owner; isakmp#929; idle; import:not set
000 #930: "conn-name-east/1x1" esp.ff5004e8@52.XX.XX.XX esp.b8805a31@10.1.1.101 tun.0@52.XX.XX.XX tun.0@110.1.1.101 ref=0 refhim=4294901761

That should get you started, so good luck and remember kids. It works for me!