Development

Atomic Deployments with CircleCI

I wrote a blog post a few months ago about using DeployBot as an atomic deploy system for your Ember apps. The atomic in atomic deployments refers to a database concept, atomicity, that requires each transaction to be “all or nothing”. In the context of a deployment, that means if something goes wrong (say you forgot to put a new dependency in your requirements.txt) the whole deployment is scrapped and your site keeps using the older, working version of your code.

When I wrote that post, we were using DeployBot to deploy entire applications, frontend and backend, at the same time. Since then, we've started to deploy our back-ends and front-ends separately using CircleCI and Ember CLI Deploy, primarily because of speed. Ember CLI Deploy is a lot faster than DeployBot. It takes 30 seconds on average to make a deployment compared to 15 minutes for DeployBot. And it supports revisions so you can rollback builds just as easily as you can with DeployBot.

That said, we could have continued to use DeployBot for back-end deployments but we choose not to for a few reasons:

We've made testing each and every commit as soon as it is made, regardless of whether or not it is getting deployed, a central part of our development process. 

Ideally you run tests manually before you push a commit, but sometimes you forget so it's helpful to make it an automatic thing. 

Atomic Deployments with CircleCI

We still wanted the ability to deploy our back-ends atomically though. Which is what this blog post is about. I am going to show you how we ended up doing DeployBot style atomic deployments with CircleCI, by walking you through the deployment of a simple Python app.

DeployBot is built for deployment, not continuous integration/delivery.

You can run tests as a user-defined command. But DeployBot’s central mission is to deploy things. We wanted to use a single tool that would test everything and deploy some things.

CircleCI has a testing integration with Github that makes it super easy for our non-technical team members to deploy their changes.

They can push their commit up to their feature specific branch, and when that little green check mark pops up they know it's safe to merge and deploy.

Atomic Deployments with CircleCI

A Script for Atomic Deployments

CircleCI differs from DeployBot in that it kind of leaves you on your own in regards to your actual deployments. Lucky for us, DeployBot wrote an awesome blog post that explains exactly how their atomic deployments work. Their strategy basically boils down to keeping multiple versions of your code on your server and pointing Apache/Nginx to the most recent one via a symbolic link. To do that, Deploybot uses a folder structure that looks like this:

application/
	current
	releases/
        	buildA/
		buildB/
		buildC/
	shared/
		environment_variables.sh

Where current is a symlink to the most recent build in the releases folder and the shared folder is used to store files that aren’t kept in source control and that’ll be used across releases (think config files and encryption keys).

We’re going to use the same strategy, but we have to implement it ourselves. We need to create a project directory on our server with two folders: releases and shared. Then we need to write a script to handle deploying our app atomically. That means that when we call our script it should:

  1. Create a new folder in releases, using the current git commit hash as the folder name

# Get current the hash of the most recent commit, we'll
# use it as our version number when we make a new folder under releases
cd /var/www/example/current && git remote update
HEAD=$(cd /var/www/example/current && git rev-parse @{u})
if [ -d "/var/www/example/releases/$HEAD" ]; then
  echo "Commit $HEAD is already deployed."
  exit 1
fi

# If the most recent commit isn't already deployed, pull it down
# and create a new virtualenv for it.
git clone git@github.com:builtbykrit/atomic-deployments-with-circleci.git /var/www/example/releases/$HEAD
cd /var/www/example/releases && virtualenv -p python3 $HEAD

2. Install any dependencies.

# Install dependencies
CURRENT=$(echo /var/www/example/releases/$HEAD)
cd $CURRENT && \
  source $CURRENT/bin/activate && \
  pip install -r $CURRENT/requirements.txt

3. Test our app to make sure the production environment is setup correctly. That way if we do something that would break the app, like add a key to our development environment variables and forget to do it in production, the deployment will fail, service won’t be interrupted and we can fix it.

# Before we update the current symlink, run tests just
# to make sure everything works on the production environment
source $CURRENT/bin/activate
python $CURRENT/manager.py test
if [ "$?" -eq "1" ]
then
  # Tests failed
  rm -rf $CURRENT
  echo "Fatal: Tests failed"
  exit 1
Fi

4. Update our symlink and restart our app.

# Tests worked, update the current symlink and restart supervisor
rm /var/www/example/current && ln -s $CURRENT /var/www/example/current
supervisorctl reload
echo "Restarting API"
sleep 5
supervisorctl stop example
supervisorctl start example

5. Clean up our releases folder, we only want to keep the last five or so.

# Only keep last 5 releases
NUM_RELEASES=$(cd /var/www/example/releases && (ls -t)|wc -l)
if [ $NUM_RELEASES -gt 5 ]; then
  cd /var/www/example/releases && (ls -t|head -n 5;ls)|sort|uniq -u|xargs rm -rf
fi

Using the Script

With a deploy script in hand we are nearly there. We just need to call it from CircleCI, using the deployment section in circle.yml:

deployment:
  master:
     branch: master
     commands:
       - ssh -i /home/ubuntu/.ssh/id_production deploy@example.com "/application/shared/atomic_deployments.sh"

To do that, we need:

  1. A user called deploy on our server, which you can create like so:
sudo useradd -d /home/deploy/ -m deploy

2. An ssh key, with no password, that we can use to login as deploy. DigitalOcean has a great walkthrough on how to set this up.Once you have generated an ssh key, you can add it to CircleCI under Project Settings > SSH Permissions. If you run into any issues they’ve written about it more thoroughly here.

3. And a copy of our deployment script on our server, in our shared folder

If you want to see the whole kit and caboodle, in the context of a real project, you can check it out on github here. Note that atomic_deployments.sh and our supervisor configuration are under the deployment folder, and circle.yml is in the project root.

Bonus

If your application is popular enough, you might run into a situation where a request starts on one version of your application and ends on another (e.g. your code accesses other files and those files get updated mid-request). You can fix that by having your server use the path your symlink references instead of using your symlink. Etsy wrote a super detailed blog post about doing that in Apache. If you are using Nginx you do the same thing by using $realpath_root instead of $document_root:

fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; 
fastcgi_param DOCUMENT_ROOT $realpath_root;
Bill Brower

Bill is the lead developer at Krit. He writes about technical stuff like how to handle big loads with Apache. If you want to talk about nerdy stuff like rap music and fantasy football drop him a line .