testing, staging, deployment processes

justin justin at justinhileman.info
Wed Sep 21 02:29:13 MDT 2011

We use this:


This also works really well:


My team actually uses a bit of a hybrid between the two. We're working
toward decoupling features from release dates, and as we do that, we
end up moving more toward the "github flow" than gitflow.

Our current flow goes something like this:

 * We have one 'production' branch that contains the codebase which is
currently live on our server.

 * We have a second 'develop' branch which contains the code currently
in QA to go live with the next release.

 * We have any number of 'feature' branches which are currently under
development. Right now that number is 40, but that's lower than usual
because we cut a release branch this morning and merged down ~45
feature branches.

We also might have:

 * Zero or one 'release/x.y.z' branches. These are a snapshot of
develop which undergoes a full manual regression test by our QA team.
A release branch is cut every two weeks, and will become our next
tagged release and 'production' branch — if and when it has no
blocking bugs left.

 * Zero or one 'hotfix/x.y.z' branches. These branches are very short
lived, and only exist as a place to write and test critical bugfixes
before merging them into production and deploying.

We have matching continuous integration servers for the 'production',
'development' and 'release' branches. Every commit against them is
automatically deployed and tested — and results in either a "1-up" or
a "Mario died" noise on my computer :)

We also have a handful of staging servers (currently between 3 and 5?)
which are used to manually test code before it goes out.

 * The primary staging server is automatically deployed from 'develop'
on a given cycle, unless there happens to be an open 'release/x.y.z'
branch, in which case it will auto-deploy the release branch instead.

 * Secondary, tertiary, etc staging servers are available for
developers, QA, and other employees to deploy and test arbitrary code

Right now, the additional staging servers are a static set, but one of
the sysadmins is working on setting up an automated EC2 pool of
staging servers, which can be deployed as needed to test features or

This is working out quite well for us, but it would be quite a bit to
set up all at once :)

-- justin

On Tue, Sep 20, 2011 at 6:34 PM, Wade Preston Shearer
<wadeshearer.lists at me.com> wrote:
> I am considering ways to improve our testing, staging, and deployment processes. We have multiple sites being worked on by multiple developers. I would like to hear how others (especially large teams) are doing it.
> Our current process works like this:
> 1. developer writes code, tests, and then commits to trunk
> 2. developer submits a push request to the "push-guy"
> 3. push-guy updates beta server (working copy of trunk) with files in request and returns note to developer
> 4. developer checks code on beta server and returns note of success
> 5. push-guy rsyncs files to staging server and returns note to developer
> 6. developer checks code on staging server and returns note of success
> 7. push-guy rysncs files to production cluster and returns note to developer
> 8. developer checks code on production server(s) and returns note of success
> This works well for a single developer, but once you have multiple developers submitting requests, testing, and verifying, it gets really hard to manage. And, it gets especially difficult when you have certain files that are found buggy and shouldn't be pushed with the rest of the updates.
> To help clarify things and provide ways to roll back easier, we have been considering implementing a beta branch and a stable tag. This would allow us to be okay with trunk being unstable and only merge over files that should be deployed. The beta server would be a working copy of the beta branch and the staging server wold be a working copy of the stable branch. We would then rsync from staging to the production cluster.
> This doesn't help much with management of things though (submission of items for deployments, testing, verifying that it's been peer-reviewed, tested, etc.) and merging adds a lot of extra steps (merge, commit, merge, commit).
> What are you doing? What systems have you heard of large teams using?
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */


More information about the PLUG mailing list