Push The Button

Photo: http://flickr.com/photos/unseelie/766346338/

How many steps does it take to get a change live on your website?

Ideally it should be a one click process.

Otherwise, when the pressure is on (i.e. when there is a bug on the site that you quickly need to fix) you’re sure to forget some critical step and make an even bigger hole for yourself.

What we called “the deployment process” changed a lot during my time at Trade Me.

In the very early days we just copied ASP scripts directly onto the production server. We only got away with this because there were not many people writing code and there were not many people using the site.

Later, as we moved to having multiple web servers which each required a copy of the code, we created a simple Windows application which copied the files from our local directory onto each of the web servers and would also execute selected SQL scripts against the production database. This was much better, but still relied on the developer doing the push to have the correct files on their local machine.

As the site got bigger there were some new complications. For a start there were more people involved. The teams responsible for testing and for maintaining the database and servers got increasingly nervous about developers having the ability to push code at any time. The code base got bigger, making it more difficult to keep in sync. The number of people using the site increased massively, making it less and less practical to just put code changes multiple times during the day. And, we also moved to using ASP.NET, which added the complication of having a build step in the process.

To address some of these issues we developed a new tool we called the “Release Manager” which hooked into source control and allowed us to package up changes so that they could be pushed to test or to production with one click (using simple NAnt scripts under the covers). This removed a lot of the complexity and stress from the process.

I’m sure it has continued to evolve since I left – if anybody from Trade Me is reading it would be interesting to hear about how you do it now.

Towards the end of my time there the test team, who had final sign-off on each release (twice per day at that point), got into the habit of queuing up ‘Push The Button’ by the Sugarbabes on the MP3 player when they were ready for changes to be deployed to production. Every time I hear that song now my pulse increases slightly at the prospect of some site changes going live!

I always thought it would be fun to wire up a proper red button to trigger the deployment, but never got the time …

If you’re interested, I wrote a little more about the tools and processes we used (as at April ’07) here:

Questions from Tim Haines, Part II

How do you manage deployment?

9 thoughts on “Push The Button”

  1. cap deploy

    I even have Capistrano set up to deploy static html sites from Subversion. The latest version also handles Git repositories so it may be time to jump on the latest SCM bandwagon ;-)

    I have also just added some Capistrano extensions so that I can now to ‘cap staging deploy’ and ‘cap production deploy’ to deploy to different servers for testing, etc. I have had too many problems in the past forgetting steps, not uploading files and generally screwing things up to not use automation.

  2. NAnt. Builds our development server and a package for live deployment. Rather than copy files over the live server files, I create a new version folder, copy the files into that and then change the IIS path reference to point to that. Minor releases get the personal touch.

  3. On TopGear.com (the new one, not the current one) we have a multi-step process:

    1. build the app (it’s .NET) using NANT (one command)
    2. build a deployment package (1-3 commands, one each for SYSTEST, UAT and LIVE)
    3. hand it off to the production people, who actually do the deployment. Sadly, we have to do it this way, but for them it’s a case of unzip then run nant. Easy.

    Was a bit of a mission to get the scripts written, but it’s paid off. What was a 1-2 day process can now be done 3-5x/day if we have to.

  4. Reminds me of an email one of the testers sent me one afternoon after I queued “Push the Button”. I went something like “Hey Adam, just FYI, we queue up Push the Button to let dev know we are ready for our morning or afternoon deploy. I hope they didn’t just deploy something…”

  5. Just like Glen, every one of my clients is a heavy user of capistrano for automated deployment. A single script to control parallel execution of commands on an entire cluster, that’s pretty cool.

    I even use it on that project I’ve been working on lately.

  6. I wrote about this a while back, mainly after my experiences in government and big business. All about making the cost of failure really really cheap.

    In most of those places, a deployment was a major multi-step (multiple chance of failure) process. Which meant… deployment happened infrequently, and was big (with greater chance of something being wrong). Which led to … lots of post-deployment analysis (whose fault was it?) meetings.

    It takes a while to sort out a simple deploy process, but it is so worth it! But one thing I havent really seen done all that well, is synchronising database updates with code deployments. How does everyone do it?

  7. Greg,

    At Trade Me we created scripts for all database changes. Stored procedure changes were included in the package and executed automatically against the database as the code changes were deployed.

    Changes to table columns, indices, etc were also scripted, but were normally manually executed (given that the site would need to be taken offline to make these sort of changes that wasn’t really a constraint).

Comments are closed.