Centralizing distributed version control

We switched over to using Mercurial about a year and a half ago (from tla/baz---don't ask), and it's worked out quite well for us. One of the key issues we ran up against is how to come up with a reasonable workflow, one that allowed people to work independently when they needed, and also included an effective central coordination point.

We ended up coordinating our development through the use of a compile daemon. The basic idea of the daemon is simple: for any daemonized tree, the compile daemon maintains two repositories, a staging repo, and a main repo. The staging repo is where you push something that you want to be considered by the compile daemon, and the main repo is where you go to grab a fresh copy of the tree. The workflow is pretty simple. The staging repo is multi-headed, which is to say that there is no unique revision that is the most up-to-date. Instead, there are a bunch of competing development heads. The compile daemon's job is to consider each of these heads in turn for inclusion into the main repo. The daemon tests a head by merging it with the current head of the main repo, compiling the resulting tree, and running unit tests. If any of those steps fail, the compile daemon gives up on that head. If all the steps succeed, than the new revision becomes the new head of the main repo.

Our approach differs from the normal suggestion for creating a primary repo, which is to use a pull model: you have someone who is responsible for pulling and merging patches from other people's trees, and thereby generating a central tree that others can pull from. But the compile-daemon workflow seems to work better for us. For one thing, there is no single sensible maintainer for some of our trees. Our primary tree contains about 800kloc, and dozens of different projects. We like to throw them all together so that the compile daemon can check that the whole damn thing builds. This disincentizes developers from breaking other people's code, since in order to get their changes in, they need to fix every type error and every broken unit test their change would lead to. The sheer size of the tree and the high rate of change makes it hard for any one person to serve as a gateway for changes to get through.

Another thing about this model is that you need an effective mix of type-checking and unit-testing to make sure that the tests run by the compile daemon are effective in weeding out broken code. It also greatly increases the incentive for programmers to write unit tests, since unit tests become a way for programmers to defend their code against breakage by others.