testing, staging, deployment processes

S. Dale Morrey sdalemorrey at gmail.com
Wed Sep 21 08:01:45 MDT 2011

A little background.  I have worked on teams ranging in size from 1
person (just me), to 5,000 in a single org.  I've held positions
ranging from entry level software developer, to lead developer, all
the way to project manager.

For most of the successful larger teams I've worked on, the process is
pretty much as you described.
The difference is subtle and has to do with the way QA is viewed in
the organization.

Three primary branches are created in the repo, called dev, test and stable.

First the team as a whole develops the acceptance criteria for the
product they are creating and writes those criteria into tests that
can verify as much functionality as possible at compile time, these
are called unit tests.  Once the tests are in place, they form the
basis of the dev and test branches.

The software team then develops source code which will pass the
compile time testing (or possibly new tests if it's found out the
acceptance criteria were incorrectly specified), then a patch is
submitted to the QA branch.

No matter how many people you have, a QA department should handle
testing the code for anything beyond the initial "compile & pray"
stage and provide detailed feedback to the developer(s) (although
generally it will be the team lead) responsible for the source code.

  If your organization is just starting out, then your entire QA
department may in fact be just one guy or gal, may even be you
yourself but you should treat it mentally as a separate job.

QA's ONLY job is to test against the documented acceptance criteria,
and either accept or reject it.  If you are good at writing your own
software tests (this includes unit testing but also other tests like
client/server dummies), then you will generally find your rejections
are limited to criteria that cannot be easily automated.  (In my
experience those failures are generally against subjective criteria.)

Even though QA can be a completely separate department not affiliated
with the dev department; I did once work for a company where the QA
department was comprised primarily of the software development team
leads and in my opinion this was the most effective setup, since if it
doesn't work, you're gonna hear about it directly from your boss.

After QA deems that your team's code is fully functional and passes
all acceptance criteria, a patch is then made and applied from the
test branch to the stable branch.  Since QA are the gatekeepers here
it's their job to make sure that this part of the process works.
Since it's an aggregate step instead of a per developer step, it's
generally successful, and the QA folks are expected to throw the dev
grunts a pizza party each time this happens :)

Now there are numerous exceptions to this, but most of the truly
successful organizations I've worked for use this or some variant.

For instance exceptionally large software products, may not have a 1
to 1 mapping between dev & qa.  Instead they isolate the functionality
of the product into various domains, each with their own dev teams.
Then there are aggregate QA teams that handle input from multiple dev
teams.  Hopefully with a final master QA team that includes beta

Honestly though, projects of that size and scope will generally be
enterprise projects that touch on more than just software.  In this
case the project will hopefully have a project manager co-ordinating
everything and facilitating communication across the vastness of the


On Wed, Sep 21, 2011 at 2:29 AM, justin <justin at justinhileman.info> wrote:
> We use this:
>    http://nvie.com/posts/a-successful-git-branching-model/
> This also works really well:
>    http://scottchacon.com/2011/08/31/github-flow.html
> My team actually uses a bit of a hybrid between the two. We're working
> toward decoupling features from release dates, and as we do that, we
> end up moving more toward the "github flow" than gitflow.
> Our current flow goes something like this:
>  * We have one 'production' branch that contains the codebase which is
> currently live on our server.
>  * We have a second 'develop' branch which contains the code currently
> in QA to go live with the next release.
>  * We have any number of 'feature' branches which are currently under
> development. Right now that number is 40, but that's lower than usual
> because we cut a release branch this morning and merged down ~45
> feature branches.
> We also might have:
>  * Zero or one 'release/x.y.z' branches. These are a snapshot of
> develop which undergoes a full manual regression test by our QA team.
> A release branch is cut every two weeks, and will become our next
> tagged release and 'production' branch — if and when it has no
> blocking bugs left.
>  * Zero or one 'hotfix/x.y.z' branches. These branches are very short
> lived, and only exist as a place to write and test critical bugfixes
> before merging them into production and deploying.
> We have matching continuous integration servers for the 'production',
> 'development' and 'release' branches. Every commit against them is
> automatically deployed and tested — and results in either a "1-up" or
> a "Mario died" noise on my computer :)
> We also have a handful of staging servers (currently between 3 and 5?)
> which are used to manually test code before it goes out.
>  * The primary staging server is automatically deployed from 'develop'
> on a given cycle, unless there happens to be an open 'release/x.y.z'
> branch, in which case it will auto-deploy the release branch instead.
>  * Secondary, tertiary, etc staging servers are available for
> developers, QA, and other employees to deploy and test arbitrary code
> on.
> Right now, the additional staging servers are a static set, but one of
> the sysadmins is working on setting up an automated EC2 pool of
> staging servers, which can be deployed as needed to test features or
> integration.
> This is working out quite well for us, but it would be quite a bit to
> set up all at once :)
> -- justin
> On Tue, Sep 20, 2011 at 6:34 PM, Wade Preston Shearer
> <wadeshearer.lists at me.com> wrote:
>> I am considering ways to improve our testing, staging, and deployment processes. We have multiple sites being worked on by multiple developers. I would like to hear how others (especially large teams) are doing it.
>> Our current process works like this:
>> 1. developer writes code, tests, and then commits to trunk
>> 2. developer submits a push request to the "push-guy"
>> 3. push-guy updates beta server (working copy of trunk) with files in request and returns note to developer
>> 4. developer checks code on beta server and returns note of success
>> 5. push-guy rsyncs files to staging server and returns note to developer
>> 6. developer checks code on staging server and returns note of success
>> 7. push-guy rysncs files to production cluster and returns note to developer
>> 8. developer checks code on production server(s) and returns note of success
>> This works well for a single developer, but once you have multiple developers submitting requests, testing, and verifying, it gets really hard to manage. And, it gets especially difficult when you have certain files that are found buggy and shouldn't be pushed with the rest of the updates.
>> To help clarify things and provide ways to roll back easier, we have been considering implementing a beta branch and a stable tag. This would allow us to be okay with trunk being unstable and only merge over files that should be deployed. The beta server would be a working copy of the beta branch and the staging server wold be a working copy of the stable branch. We would then rsync from staging to the production cluster.
>> This doesn't help much with management of things though (submission of items for deployments, testing, verifying that it's been peer-reviewed, tested, etc.) and merging adds a lot of extra steps (merge, commit, merge, commit).
>> What are you doing? What systems have you heard of large teams using?
>> /*
>> PLUG: http://plug.org, #utah on irc.freenode.net
>> Unsubscribe: http://plug.org/mailman/options/plug
>> Don't fear the penguin.
>> */
> --
> http://justinhileman.com
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */

More information about the PLUG mailing list