Zero downtime Rails redeploys with Unicorn

Like any self-respecting Ruby hipster I am using Unicorn as my app server in production. Unlike most Ruby hipsters my deployment process is pretty manual at the moment. While the high-touch manual deployment that I am currently doing is far from ideal long term, short term its giving me a close up look at the care and feeding of a production Rails app. Think of it as wanting to get my hands dirty after years of being coddled by Heroku. :)

Much ado has been made of how unixy Unicorn is, and one of the ways that manifests itself is how Unicorn uses signals to allow you to talk to a running server process. What has been interesting about this has been a reintroduction to the “kill” command. Its pretty common to know that “kill -9 1234” is a quick way to kill process 1234 but it turns out that the kill command has much more happening. The mysterious -9 option is significantly less mysterious once know that kill can send ANY signal, and finally look at the list of options:

mike@sleepycat:~☺  kill -l
 1) SIGHUP	 2) SIGINT	 3) SIGQUIT	 4) SIGILL	 5) SIGTRAP
 6) SIGABRT	 7) SIGBUS	 8) SIGFPE	 9) SIGKILL	10) SIGUSR1
11) SIGSEGV	12) SIGUSR2	13) SIGPIPE	14) SIGALRM	15) SIGTERM
16) SIGSTKFLT	17) SIGCHLD	18) SIGCONT	19) SIGSTOP	20) SIGTSTP
21) SIGTTIN	22) SIGTTOU	23) SIGURG	24) SIGXCPU	25) SIGXFSZ
26) SIGVTALRM	27) SIGPROF	28) SIGWINCH	29) SIGIO	30) SIGPWR
31) SIGSYS	34) SIGRTMIN	35) SIGRTMIN+1	36) SIGRTMIN+2	37) SIGRTMIN+3
38) SIGRTMIN+4	39) SIGRTMIN+5	40) SIGRTMIN+6	41) SIGRTMIN+7	42) SIGRTMIN+8
43) SIGRTMIN+9	44) SIGRTMIN+10	45) SIGRTMIN+11	46) SIGRTMIN+12	47) SIGRTMIN+13
48) SIGRTMIN+14	49) SIGRTMIN+15	50) SIGRTMAX-14	51) SIGRTMAX-13	52) SIGRTMAX-12
53) SIGRTMAX-11	54) SIGRTMAX-10	55) SIGRTMAX-9	56) SIGRTMAX-8	57) SIGRTMAX-7
58) SIGRTMAX-6	59) SIGRTMAX-5	60) SIGRTMAX-4	61) SIGRTMAX-3	62) SIGRTMAX-2
63) SIGRTMAX-1	64) SIGRTMAX	

So with that knowledge lets send some signals to Unicorn to get it serving up the latest version of our code. First we need its process id. We are really just interested in the process id of the master process which we can see is 26465:

mike@sleepycat:/myapp$ ps aux | grep unicorn
503       7995   0:00 grep unicorn
503      26465   0:07 unicorn_rails master -c config/unicorn.rb --env production -D
503      26498   0:11 unicorn_rails worker[0] -c config/unicorn.rb --env production -D
503      26502   2:37 unicorn_rails worker[1] -c config/unicorn.rb --env production -D
503      26506   0:06 unicorn_rails worker[2] -c config/unicorn.rb --env production -D
503      26510   0:06 unicorn_rails worker[3] -c config/unicorn.rb --env production -D
503      26514   0:06 unicorn_rails worker[4] -c config/unicorn.rb --env production -D
503      26518   0:06 unicorn_rails worker[5] -c config/unicorn.rb --env production -D
503      26522   0:06 unicorn_rails worker[6] -c config/unicorn.rb --env production -D
503      26526   0:07 unicorn_rails worker[7] -c config/unicorn.rb --env production -D
503      26530   0:07 unicorn_rails worker[8] -c config/unicorn.rb --env production -D
503      26534   0:06 unicorn_rails worker[9] -c config/unicorn.rb --env production -D
503      26538   0:09 unicorn_rails worker[10] -c config/unicorn.rb --env production -D
503      26542   0:07 unicorn_rails worker[11] -c config/unicorn.rb --env production -D
503      26546   0:07 unicorn_rails worker[12] -c config/unicorn.rb --env production -D
503      26550   0:08 unicorn_rails worker[13] -c config/unicorn.rb --env production -D
503      26554   0:10 unicorn_rails worker[14] -c config/unicorn.rb --env production -D
503      26558   0:08 unicorn_rails worker[15] -c config/unicorn.rb --env production -D
503      26562   0:05 unicorn_rails worker[16] -c config/unicorn.rb --env production -D
503      26566   0:08 unicorn_rails worker[17] -c config/unicorn.rb --env production -D
503      26570   0:07 unicorn_rails worker[18] -c config/unicorn.rb --env production -D
503      26574   0:06 unicorn_rails worker[19] -c config/unicorn.rb --env production -D         

Since I have just pulled down some new code I want to restart the master process. I can get Unicorn to launch an new master process by sending the master process the USR2 signal. After that you can see that there is now a new master (7996) with its set of workers and the old master (26465) and its set of workers

mike@sleepycat:/myapp$ kill -USR2 26465
mike@sleepycat:/myapp$ ps aux | grep unicorn
503       7996  0:07 unicorn_rails master -c config/unicorn.rb --env production -D
503       8035  0:00 unicorn_rails worker[0] -c config/unicorn.rb --env production -D
503       8038  0:00 unicorn_rails worker[1] -c config/unicorn.rb --env production -D
503       8041  0:00 unicorn_rails worker[2] -c config/unicorn.rb --env production -D
503       8044  0:00 unicorn_rails worker[3] -c config/unicorn.rb --env production -D
503       8046  0:00 unicorn_rails worker[4] -c config/unicorn.rb --env production -D
503       8050  0:00 unicorn_rails worker[5] -c config/unicorn.rb --env production -D
503       8052  0:00 unicorn_rails worker[6] -c config/unicorn.rb --env production -D
503       8056  0:00 unicorn_rails worker[7] -c config/unicorn.rb --env production -D
503       8059  0:00 unicorn_rails worker[8] -c config/unicorn.rb --env production -D
503       8062  0:00 unicorn_rails worker[9] -c config/unicorn.rb --env production -D
503       8064  0:00 unicorn_rails worker[10] -c config/unicorn.rb --env production -D
503       8069  0:00 unicorn_rails worker[11] -c config/unicorn.rb --env production -D
503       8073  0:00 unicorn_rails worker[12] -c config/unicorn.rb --env production -D
503       8075  0:00 unicorn_rails worker[13] -c config/unicorn.rb --env production -D
503       8079  0:00 unicorn_rails worker[14] -c config/unicorn.rb --env production -D
503       8082  0:00 unicorn_rails worker[15] -c config/unicorn.rb --env production -D
503       8085  0:00 unicorn_rails worker[16] -c config/unicorn.rb --env production -D
503       8088  0:00 unicorn_rails worker[17] -c config/unicorn.rb --env production -D
503       8091  0:00 unicorn_rails worker[18] -c config/unicorn.rb --env production -D
503       8094  0:00 unicorn_rails worker[19] -c config/unicorn.rb --env production -D
503       8156  0:00 grep unicorn
503      26465  0:07 unicorn_rails master (old) -c config/unicorn.rb --env production -D
503      26498  0:11 unicorn_rails worker[0] -c config/unicorn.rb --env production -D
503      26502  2:37 unicorn_rails worker[1] -c config/unicorn.rb --env production -D
503      26506  0:06 unicorn_rails worker[2] -c config/unicorn.rb --env production -D
503      26510  0:06 unicorn_rails worker[3] -c config/unicorn.rb --env production -D
503      26514  0:06 unicorn_rails worker[4] -c config/unicorn.rb --env production -D
503      26518  0:06 unicorn_rails worker[5] -c config/unicorn.rb --env production -D
503      26522  0:06 unicorn_rails worker[6] -c config/unicorn.rb --env production -D
503      26526  0:07 unicorn_rails worker[7] -c config/unicorn.rb --env production -D
503      26530  0:07 unicorn_rails worker[8] -c config/unicorn.rb --env production -D
503      26534  0:06 unicorn_rails worker[9] -c config/unicorn.rb --env production -D
503      26538  0:09 unicorn_rails worker[10] -c config/unicorn.rb --env production -D
503      26542  0:07 unicorn_rails worker[11] -c config/unicorn.rb --env production -D
503      26546  0:07 unicorn_rails worker[12] -c config/unicorn.rb --env production -D
503      26550  0:08 unicorn_rails worker[13] -c config/unicorn.rb --env production -D
503      26554  0:10 unicorn_rails worker[14] -c config/unicorn.rb --env production -D
503      26558  0:08 unicorn_rails worker[15] -c config/unicorn.rb --env production -D
503      26562  0:06 unicorn_rails worker[16] -c config/unicorn.rb --env production -D
503      26566  0:08 unicorn_rails worker[17] -c config/unicorn.rb --env production -D
503      26570  0:07 unicorn_rails worker[18] -c config/unicorn.rb --env production -D
503      26574  0:06 unicorn_rails worker[19] -c config/unicorn.rb --env production -D

s
Now I want to shutdown the old master process and its workers. I can do that with the QUIT signal:

mike@sleepycat:/myapp$/myapp$ kill -QUIT 26465
mike@sleepycat:/myapp$/myapp$ ps aux | grep unicorn
503       7996  0:07 unicorn_rails master -c config/unicorn.rb --env production -D
503       8035  0:00 unicorn_rails worker[0] -c config/unicorn.rb --env production -D
503       8038  0:00 unicorn_rails worker[1] -c config/unicorn.rb --env production -D
503       8041  0:00 unicorn_rails worker[2] -c config/unicorn.rb --env production -D
503       8044  0:00 unicorn_rails worker[3] -c config/unicorn.rb --env production -D
503       8046  0:00 unicorn_rails worker[4] -c config/unicorn.rb --env production -D
503       8050  0:00 unicorn_rails worker[5] -c config/unicorn.rb --env production -D
503       8052  0:00 unicorn_rails worker[6] -c config/unicorn.rb --env production -D
503       8056  0:00 unicorn_rails worker[7] -c config/unicorn.rb --env production -D
503       8059  0:00 unicorn_rails worker[8] -c config/unicorn.rb --env production -D
503       8062  0:00 unicorn_rails worker[9] -c config/unicorn.rb --env production -D
503       8064  0:00 unicorn_rails worker[10] -c config/unicorn.rb --env production -D
503       8069  0:00 unicorn_rails worker[11] -c config/unicorn.rb --env production -D
503       8073  0:00 unicorn_rails worker[12] -c config/unicorn.rb --env production -D
503       8075  0:00 unicorn_rails worker[13] -c config/unicorn.rb --env production -D
503       8079  0:00 unicorn_rails worker[14] -c config/unicorn.rb --env production -D
503       8082  0:00 unicorn_rails worker[15] -c config/unicorn.rb --env production -D
503       8085  0:00 unicorn_rails worker[16] -c config/unicorn.rb --env production -D
503       8088  0:00 unicorn_rails worker[17] -c config/unicorn.rb --env production -D
503       8091  0:00 unicorn_rails worker[18] -c config/unicorn.rb --env production -D
503       8094  0:00 unicorn_rails worker[19] -c config/unicorn.rb --env production -D
503       8161  0:00 grep unicorn

So now we have a Unicorn serving up the latest version of our code without dropping a single request. Really slick.
Ryan Bates has a screencast that has a broader look at the subject of zero downtime deployments, automated with Capistrano (probably a more sustainable approach), but if you look closely you will see these signals lurking in the code.

If you are interested in digging into more Unix fundamentals (from a Rubyist’s perspective!) I would recommend checking out Jesse Storimer’s books.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s