Causes Tech: Make your deploys blazing fast by making them git-aware
Posted Aug 23, 2011 by Adam Derewecki
Adam is a software engineer here at Causes and has the best personal bio on our team page.
Fast deploys are important for keeping up overall momentum on a programming team. If an entire deploy takes 40 minutes, friction is high and deploys happen less frequently — meaning that “finished” code sits around longer waiting to go into production. The holy grail of code deployment is continuous deployment, where every change is automatically tested and (if tests show no problems) pushed into production. To reach continuous deployment, you need fast tests and fast deploys. Since we already have fast tests, we decided to take some time to speed up our deployment process.
Our existing deployment process is one that I’ve seen with several engineering teams:
1. Create a tarball of the code revision you want to deploy
2. Transfer (scp/rsync) that tarball to a deployment server
3. Transfer the tarball from the deployment server to each appserver
4. Extract the tarball to a directory containing deployments
5. Switch the `current` symlink in deployments/ to point to the new directory
6. Perform a rolling restart of the appservers
There are a few bottlenecks in this process. First, you need a fast connection from your development machine to the deployment server (usually the former is home/office and the latter is in a colo) because you’re sending an entire copy of your codebase over the wire. If your codebase is a few megs, this isn’t a big deal — but as you start approaching several hundred megabytes, this becomes a major painpoint. And why do you need to send everything, anyway? The diff you’re deploying is probably only a few kilobytes different from what’s currently in production, and git knows how to apply diffs pretty well. A better method:
1. Create a clone of your repo on each appserver (this only needs to happen once)
2. Tell each appserver to fetch and reset –hard to the specified revision (be careful with submodules)
3. Use rsync to make a copy of the repo to the deployments/ directory, omitting the .git directory
4. Switch the `current` symlink in deployments/ to point to the new directory
5. Perform a rolling restart of the appservers
After building this logic into our deployment process, our average deploys went from 5-20 minutes to 2-3 minutes. Currently, the slowest part of the deploy process is the rake task that determines if a migration needs to be run (most of that is Rails startup time) and the rolling restart, which is slow on purpose (we don’t want all appservers to simultaneously be unable to serve traffic). We’ve put together a reference implementation of the code that needs to be run on each appserver. This should be used as a guideline for building your own, and is probably not robust enough as-is to use in your production environment (but it does cover submodules, which is something that seems to be lacking on other write-ups of this topic).