One of the cool things at LifeChurch.tv is that Team Digerati once a month gets together and has someone on the team talk about some cool technology and share what they know.
Last month, we had three such talks, and I was one of them. I spoke at length about HAProxy, how we use HAProxy and some of the challenges we've had with HAProxy. If you have any interest, the video and slides are now up here.
Just wanted to send a shout out to any system engineers/administrators that might be reading this that my organization, LifeChurch.tv, is currently hiring another fulltime Linux sysadmin.
If you want to help change the world with us, we want to hear from you. Check it out.
One of the pain points I've discovered recently with the new version of OS X Lion is how it handles (or, rather, doesn't handle) .local domains in /etc/hosts. We use .local domains heavily in our development environment, and they no longer work in Lion. All .local domains are technically reserved for Multicast DNS (Bonjour), and Lion no longer looks in /etc/hosts first when you try to lookup a .local domain. Instead, it asks MDNS first, waits out the 5 second timeout, then checks /etc/hosts.
There are several ways around this. You could install dnsmasq, though I think (but am not sure) it might have the same issues with being checked only after the 5 second mdns timeout. You could just start using an alternative TLD, like .dev.
But if you have to use .local, the most elegant solution I've found is the dscl utility. Using it is very straightforward. To add a host called mydev.local and point it to the localhost, just do this:
sudo dscl localhost -create /Local/Default/Hosts/mydev.local IPAddress 127.0.0.1
To see all the currently defined hosts and their IPs
sudo dscl localhost -list /Local/Default/Hosts IPAddress
And to remove a host:
sudo dscl localhost -delete /Local/Default/Hosts/mydev.local
Overall, pretty straightforward and works well. I still would prefer to be able to edit /etc/hosts instead, but this is a better alternative to having to rename all our .local servers.
How do you migrate a large and live running Postgresql instance to 9.0? That's one thing I was just recently tasked with doing, and there are several ways to go about this. Probably the most straightforward of ways is to upgrade from your distribution's Postgres package and using the built in upgrade utility to do a single server upgrade. It's relatively quick and easy, and I strongly suggest going down this route if you have the chance.
However, I chose to handle our upgrade a little differently. Some of the complicating factors include:
Our architecture looks something like this:
Each app server connects to a pgbouncer server which pools those connections and only makes a handful of connections back to the database server. Because connections within Postgres can be rather expensive, using a pooler like pgbouncer really helps us when we may have thousands of PHP processes, each needing data from the database server. PGBouncer is also great because it can do automatic failover if you have multiple database servers and can move connections to a new, healthy master tranparently to each PHP process.
This feature comes in really handy when you want to do a Postgres update. We had two database servers, psqlA and psqlB. psqlA is the master and runs the legacy CentOS installation with the older Postgres version. psqlB is the redundant backup server. I decided to do all the upgrades on psqlB -- it got larger harddrives, and was updated to Postgres 9. Then, using Londiste's statement based replication, I began replicating data from psqlA's 8.3 install to psqlB's 9.0 install. It looks something like this:
In theory, there is no difference between theory and practice. But in practice there is. And the same was true here. :) It wasn't quite as clean as imagined.
There were a few things that probably could have gone better. First of all, I wanted to be in full control of where database queries went, so I wasn't using pgbouncer's built in failover options. Instead, I had two config files, psqlA and psqlB, so I could move traffic back and forth as necessary. This proved to not be perfectly smooth. I was probably doing something wrong, but I ended up restarting pgbouncer to break old connections to move traffic to the new server. In the end, not a big deal, but it added a few seconds of a hard stop while the PHPs reconnected to the new pgbouncer.
The second reason I ended up putting a hardstop was because of replication. Because replication will always lag a few seconds, I ended up stopping traffic for a few seconds to let any lingering data replicate from psqlA to psqlB. Then when that happened, I stepped in and broke replication completely so that there would be no funny business if something went wrong and some traffic ended up on psqlA by accident. I shutdown psqlA's Postgres instance, brought up PGBouncer, and watched traffic start to flow to psqlB. Total downtime amounted to about 20-30 seconds on the database side, but it caused a slight backup on the app servers that lasted for about 45-120 seconds.
Unfortunately, it more than one upgrade attempt to get that result. Each upgrade went smoothly, and things were looking very good for the first few minutes, but within about 2-5 minutes of switching to psqlB, we found we were quickly running out of connections within Postgres.
Turns out, despite months of testing Postgres 9 in dev and staging, we missed that a few queries were taking much longer to execute. It was hardly noticeable to use because the query went from like .09 to .15 seconds or something like that, just enough that our testing didn't notice a difference. But when put in production, with hundreds of requests per second, caused just enough of a backup to be an issue. We ended up failing back to psqlA, investigated the issue, fixed it, and the next upgrade went much smoother. Turns out, that one of the optimization changes in PG9.0 actually makes the optimizer worse for a particular query we used a lot. Fortunately, it was an extremely simple fix that made the optimizer perform better.
Overall, it was a pretty smooth experience, and except for the minor hiccup in having to failback, the whole process was barely noticeable. I'm quite thankful for the time I took to plan out the whole upgrade process, especially having a reliable and trusted fallback position. But in the future, I know we will be paying even closer attention to even the slightest of performance deltas.
Previously, I’ve shared how I build custom Debian packages based on existing Debian packages. But sometimes the package you want to build is not actually provided by Debian, or you just want to start from scratch. Fortunately, that’s almost easier to set up! Here’s the short and simple way I do it.
Setting Up
First, let’s set up our directories:
mkdir ~/packages/php/core
Second, let’s download the source. Say PHP:
wget http://us3.php.net/get/php-5.3.5.tar.gz/from/this/mirror -o php-5.3.5.tar.gz
Unzip it
tar -xvzf php-5.3.5.tar.gz && cd php-5.3.5
Make Special debian/ Directory
Run dh_make to build the debian/ directory
dh_make —createorig
At this point you can edit any file in the debian/ directory to tweak or change anything you need. (like runtime or build dependencies) I usually like to change the version in debian/changelog to include my own version number so when I run dpkg -l I can tell it’s my package that’s installed, and which one. I like to add the format -AJB.0 to the end of the upstream version.
Build the package
Now, run
dpkg-buildpackage
This actually builds the software. If there’s a failure, you probably will need to download headers or something for a dependency. You should update the dependencies list in debian/control with the new package so that it’s used.
Finished!
Once dpkg-buildpackage does its thing, you’re technically done and good. However, you probably will want to verify that your package’s dependencies are correctly set and that the package can install in a clean Debian environment without any trouble. For this, pbuilder is very useful.
There is, of course, significantly more to do to completely finish building a true Debian package, and much of that is controlled in the debian subdirectory. But this should get you off to a good start.
You probably heard that Google has recently opened two-step verification for all Google accounts. I’m still waiting for it to be activated for my main Google account, but this afternoon I set it up for my Google Apps account.
I love that Google has implemented it across the board for all Google apps. I’m sure a large part of the motivation was making Gmail more secure after the attempted Chinese hacking incident last year. Many of us in IT have used and administered RSA SecurID infrastructure, especially for VPNs. Google is taking advantage of the ubiquity of smart phones to open this same level of security across the landscape. Obviously, it does add a level of hassle, but I think Google has done a great job making the process easy, simple, clear and safe. You get a list of 10 backup PINs to save, and you can enter your phone number so Google can call or text you with a code.
The really cool thing is that Google has released a module for PAM that allows you to use exactly the same infrastructure for SSH and all other Unix services you might be running.
I wanted to play with it and set it up in my Debian sid VM just to see how the process works. It’s pretty easy.
First, you just need to install the PAM headers, sudo and mercurial if you don’t them it already:
apt-get install mercurial libpam0g-dev sudo
Next, we need to download the module. The easiest way is just to use mercurial to clone the source from Google Code:
hg clone https://google-authenticator.googlecode.com/hg/ google-authenticator/
Go ahead and build/install it:
cd google-authenticator/libpam make install
Activate it within PAM for sshd by adding “auth required pam_google_authenticator.so” to /etc/pam.d/sshd. You’ll also need to set ChallengeResponseAuthentication to yes in /etc/ssh/sshd_config and restart ssh.
Now, all you need to do is download the Google Authenticator app for your phone (available for iPhone, Android and BlackBerry) and generate the codes for your user by running google-authenticator.
It will give you an URL you can open in your browser to view a QR code. Scan the QR code on your phone and, tada, your account will show up within the Google authenticator app. You will also be given a list of emergency scratch codes. Save these in a safe place in case you lose your phone.
Now try logging in to your SSH account again. After typing your password, you will be prompted for the latest verification code. Get it off your phone, type it in correctly, and you will be granted access. If you have SSH keys setup, you will not have to enter a verification code at all.
Opie has been around forever and I don’t think it has ever been extremely widespread. Will the added convenience and security of using the mobile app to generate a time based key change all that? Maybe. But I think it’s more likely that RSA won’t be selling as many SecurIDs in the future. :)
I definitely think there is some potential in using OTPs with an SSH gateway and SSH key forwarding. Whenever I look to see all the brute force attacks against any public server, I always think about locking SSH away on a private interface and using an SSH gateway to control access and monitor potential attacks. I could see requiring only password (plus OTP) logins to your gateway, then allowing only (forwarded) key logins to destination servers. To actually successfully login and cause damage to one of my servers, I would have to steal your password and your phone to get into the gateway, then steal your key to get into the remote server. That kind of security is extremely resilient, and very attractive to this paranoid sysadmin.
I’m sure there are probably a couple of weak areas you’d have to really think through, such as session jacking on the gateway, but with a little thought, you could really increase security by diversifying the number of ways you have to authenticate. By having the OTP exist at the gateway level, you would only need to set up your OTP account once, and you could stay connected to the gateway throughout the day, minimizing the amount of hassle users will have in adopting OTP accounts.
Now that OTP passwords are becoming more mainstream with Google’s push, will you be doing anything different with your infrastructure?
In the past, I’ve shared how we use HAProxy to help increase uptime and distribute load among any number of web servers.
HAProxy is an amazing tool, and I can’t thank the author, Willy Tarreau, enough for making it available and open source.
Of course, while HAProxy has proven to be extremely stable in production and we’ve never had an issue with it at all, it does present its own single point of failure. If your HAProxy server dies, everything that relies on HAProxy will go down with it. While I am not at all concerned about HAProxy itself crashing, I am concerned about a gamma ray hitting the CPU at just the wrong moment and nuking the whole thing.
Let’s take our infrastructure a level higher. Let’s add some redundancy on the HAProxy level to ensure that even if we nuke the front line HAProxy box, nobody’s the wiser.
We’re going to do that with a really cool tool called Pacemaker. My basic idea is that I’m going to take an IP address, and a cluster of servers. Each server will run Pacemaker and be a part of a Pacemaker cluster. I tell pacemaker that I want my IP address and HAProxy to always be available, and Pacemaker will start and stop both IPs and HAProxy on all of the different nodes to ensure it is always available.
It’s actually pretty simple, but there’s nothing cooler than pinging an HA (high availability) IP, kicking over the server it’s sitting on and watching Pacemaker instantly move the services to a different server. In my testing, I only ever lost at most 1 packet as I was pinging an IP during switchover. That just makes me so happy inside.
Enough theory, let’s dive into how to make this wonder work.
If you are on Debian (and have backports enabled), you just need to install corosync and pacemaker to get it set up and going:
apt-get install corosync pacemaker -t lenny-backports
Run this on every node in your planned cluster.
Corosync is, from my understanding, the layer that communicates with all the different nodes. It ensures that each node has a valid copy of the configuration, and it helps Pacemaker know the status of the cluster. There are other choices for this layer, but Corosync seems to be the most actively developed and mature choice currently available.
Next, we need to make sure that we have some encryption keys for Corosync to communicate with all the nodes.
corosync-keygen
Only run this on one node. Copy this key (/etc/corosync/authkey) to all the other planned nodes, and ensure permissions are set correctly. (chown root:root /etc/corosync/authkey; chmod 400 /etc/corosync/authkey)
Next, you’ll want to configure the corosync config file. The example file (in /etc/corosync/corosync.conf.example) is perfect to start out, you’ll probably only need to adjust the interface bindnetaddr value and multi-cast address. In many cases, especially if you are hosting with another provider, you can’t use multicasting. Fortunately, corosync supports broadcast just fine.
You’ll need to do each of the following steps on each node. Here’s what works for me:
interface {
ringnumber: 0
bindnetaddr: 192.168.10.0
broadcast: yes
mcastport: 5405
}Note that bindnetaddr needs to be your subnet ID for broadcast to work correctly. (for some reason, I couldn’t find clear documentation on this point when I was setting up my cluster.)
At the end, you’ll also want to add this section so corosync starts pacemaker:
service {
name: pacemaker
ver: 0
}You’ll also need a log directory, or you’ll get a cryptic error like “parse error in config”
mkdir /var/log/cluster/
Go ahead and start corosync on each node.
/etc/init.d/corosync start
With any luck, you will have your first Pacemaker high availability cluster up and running! Hooray! You can verify the status of all your nodes by running crm_mon. If some of your nodes are not listed, you will want to go back and make sure everything is setup properly.
So now that we have our cluster up and running, we can play around with the configuration and tell it what we want to make highly available across all nodes. For that, we’re going to use the crm command. If you have ever used Cisco IOS commands, you will feel at home with crm. A lot of their navigational structure really reminds me of IOS.
So let’s create a new config!
Type ‘configure’ to enter configuration mode. verify helps make sure the config is valid, show will show you the current configuration, help will show you other commands you can run. Let’s make a new configuration, and get cracking.
crm(live)configure# cib new config1
Config1 is just a name. We’re making a new configuration that we can work on that won’t actually change anything in production until we commit it and switch live to config1. This allows us to tweak and make complex dependency changes without breaking anything in the cluster.
Let’s go ahead and configure an IP address we want to make highly available. This IP should not be already assigned or setup on any existing hosts. This IP needs to be routable to all the nodes in your cluster. If you are in a hosting environment, you will need to get an IP from your provider.
So with our planned HA IP in mind, it’s just a simple matter of giving it to Pacemaker:
crm(config1)configure# primitive failover-ip ocf:heartbeat:IPaddr2 params ip=192.168.1.1 cidr_netmask=32 op monitor interval=1s
Pretty straight forward, just give it the IP address, and tell it to monitor this IP once a second.
Now we’ll make sure the config looks good, verify it, and commit it to live:
crm(config1)configure# show crm(config1)configure# verify crm(config1)configure# end crm(config1)# cib use live crm(live)# cib commit config1
Congratulations! You’ve just saved your configuration to all nodes in the cluster and made it live.
If you issue status, you should see that your IP address has started, and it will tell you on which node it started.
One thing you probably will want to do is increase the resource “stickiness”. By default, Pacemaker assumes the cost of switching a resource is 0. This just means that Pacemaker thinks there are no penalties for moving a resource around, and will do so any time a node goes down or becomes available. If you take down the node the IP was assigned, Pacemaker automatically moves it to the other node. If the original node comes up again, Pacemaker will go ahead and move the IP back to the node, even if nothing is wrong on the new node. Since we probably don’t want services moving around unless totally necessary, we will set the resource stickiness so Pacemaker will only move a service when necessary:
configure rsc_defaults resource-stickiness=100
Next, we want to make HAProxy available with the IP address. If HAProxy fails on one node, we want Pacemaker to start it (with the right IP) on another node. Pacemaker will handle starting/stopping services/IP addresses on different nodes, but it’s our responsibility to ensure each node has HAProxy installed and the correct configuration.
We’ll use the LSB (Linux Standard Base) class for HAProxy. The LSB class is for all the scripts in /etc/init.d. Note that most startup scripts, including the Debian startup script for HAProxy, are not LSB compatible. You will probably need to tweak the HAProxy startup script so it sends the right signals to Pacemaker so it will act properly. You can find out how to do that here.
In configure mode, with a new configuration, just create a new LSB primitive:
crm(config-2)configure# primitive haproxy lsb:haproxy op monitor interval="1s"
We will want to keep the failover IP and HAProxy together, it doesn’t do us any good if HAProxy runs on one machine, but the IP address is on another. So we set colocation to infinity:
crm(config-2)configure# colocation haproxy-with-public-IPs INFINITY: haproxy failover-ip
And we want to make sure that HAProxy is started after the IP address, so that we can bind to the IP:
crm(config-2)configure# order haproxy-after-IP mandatory: failover-ip haproxy
Do be careful when you set INFINITY with your colocation. If both can’t start, neither will start. I had an issue with my cluster where I had an error in my stunnel configuration on only one server, and I set stunnel and haproxy both to prefer each other with INFINITY preference, and when stunnel couldn’t start, Pacemaker shut down HAProxy even though HAProxy was fine alone. I switched the preference to 200, so that Pacemaker would know I prefer HAProxy and stunnel to run together, but that I don’t consider it a failure if both can’t run for some reason.
Now commit your changes to live and your HAProxy setup will automatically distribute itself across several servers for much better uptime, and no single point of failure! Hooray!
You should always test your clustering setup and make sure it’s acting the way you expect it to. Rip out the power or ethernet cable, shutdown corosync service, mess up your HAProxy config and reboot. Test several different failures and make sure that Pacemaker responds the way you expect it to. Make sure that each node can start up all the necessary services. Make sure that your config isn’t doing anything dumb that will actually worsen your reliability rather than improve it.
Then sleep like a baby at night because you have just removed a significant single point of failure! :)
One of the coolest tools I’ve discovered in my work with Puppet is Augeas. Augeas takes configuration files, and turns all the settings into a tree structure that you can manipulate like a file-system. Augeas takes disparate config files (ini, key/value, Apache, etc) and, in a way, turns them into an api. How cool is that!?
Here’s something I worked on just this morning. I have a bunch of rsyncd servers, and I want to add a new section to each server. Ultimately, I will do this in Puppet, but I don’t want to specify a general /etc/rsyncd.conf file, because each host has other entries that are going to be different. Augeas is the perfect tool for this job, because I can tell it exactly what lines I want to add and where, and it won’t change the rest of the file. The Puppet file tool is a chainsaw, while Augeas is a precise surgical instrument.
Installation
First, you’re going to need the Augeas tool set. This includes augtool, which lets you interactively edit config files. If you’re on Debian (and have backports setup), all you need to do is apt-get install augeas-lenses augeas-tools libaugeas-ruby1.8 libaugeas0 -t lenny-backports.
Interactive Shell
Before writing my Puppet Augeas definition, I like to test my changes interactively with augtool to make sure it works properly. So fire up augtool, and you’ll be presented with the Augeas interactive shell.
First, a couple pointers. help will show you help, and you can always type help to get info about a particular command. Within Augeas, you have a pseudo-filesystem accessible from /. You can ls /, and you will see the trees Augeas sets up. We’ll mostly use /files, which gives us access to Augeas compatible files in the local filesystem.
Augeas doesn’t let you edit any random file. Augeas needs to understand the syntax of the file and how it is structured, and for this, it uses files called Augeas lenses. You can download Augeas lenses, and can write your own. Fortunately, most system config files have a lens pre-written. The only files you will see listed in augtool ls print-outs are files that Augeas has a lens for. So if you’re trying to edit a file that should show up in /files/etc/bobsconfig.cnf, and it’s not showing up—you probably need to find or make a lens for Augeas to understand the format.
Now, let’s get to work editing our rsyncd.conf file. Fortunately, Augeas has a lens for it already, so it shows up. If we ls /files/etc/rsyncd.conf/, we’ll see that it shows up as well as the contents of the file.
It’s Full of Trees!
augtool> ls /files/etc/rsyncd.conf/ .anon/ = (none) home/ = (none)
.anon is just the first block in the rsyncd.conf file with the pid file, max connections, etc. It doesn’t belong to a server block, so Augeas just assigns it the name .anon so we have a place to put that stuff.
The home/ entry is a server block I have setup that’s called home, if I dig deeper into it, you can see all the settings I have setup for it:
augtool> print /files/etc/rsyncd.conf/home/ /files/etc/rsyncd.conf/home /files/etc/rsyncd.conf/home/comment = "Allow my home folder to be backed up to the backup server" /files/etc/rsyncd.conf/home/path = "/home/me" /files/etc/rsyncd.conf/home/read only = "yes" /files/etc/rsyncd.conf/home/list = "yes" /files/etc/rsyncd.conf/home/hosts allow = "10.11.12.13"
This shows how Augeas is making this config file into a tree, and you can begin to see how Augeas lets you interact with config files.
Setting Values
Now let’s create our own new section in the config file. For that, we’re going to use the ins command.
augtool> ins var-dub after /files/etc/rsyncd.conf/.anon/
You won’t get any output back, but let’s start populating some values
set /files/etc/rsyncd.conf/var-dub/comment "/var/www on my leet new server" set /files/etc/rsyncd.conf/var-dub/path '/var/www' set '/files/etc/rsyncd.conf/var-dub/read\ only' yes set /files/etc/rsyncd.conf/var-dub/list yes set '/files/etc/rsyncd.conf/var-dub/hosts\ allow' '10.11.12.13'
Now let’s print it so we know it has all the values we need:
augtool> print /files/etc/rsyncd.conf/var-dub/ /files/etc/rsyncd.conf/var-dub /files/etc/rsyncd.conf/var-dub/comment = "/var/www on my leet new server" /files/etc/rsyncd.conf/var-dub/path = "/var/www" /files/etc/rsyncd.conf/var-dub/read only = "yes" /files/etc/rsyncd.conf/var-dub/list = "yes" /files/etc/rsyncd.conf/var-dub/hosts allow = "10.11.12.13"
Yup. Looks good to me. Let’s go ahead and save the file and check to see what our new config file looks like:
augtool> save Saved 1 file(s) augtool> quit peter:~# cat /etc/rsyncd.conf max connections= 20 log file= /var/log/rsync.log timeout= 300 [var-dub] comment= /var/www on my leet new server path= /var/www read only= yes list= yes hosts allow= 10.11.12.13
Wrapping Up
So there you have it, a fast and easy way to manipulate config files in a programmatic way. This works great with the Augeas module for Puppet. It’s also great for automating changes that can only be made in a config file. For example, I have some auto-scaling tools setup that will automatically add and remove virtual machines within haproxy’s load balancer.
I just use an haproxy Augeas lens and use Augeas to manipulate the config file in exactly the same way we did with rsyncd’s config. In the past I would have used a template with sed, awk and grep to do the same thing, but Augeas is much easier. It’s also safer because Augeas understands the file format, and won’t write out a config that is syntactically invalid. (but you are still responsible for making sure the config does what you want it to!)
Further Reading
