Controlled anarchy, was Re: Data.* collections maintenance
carette at mcmaster.ca
Mon Oct 24 12:17:50 EDT 2005
Simon Marlow wrote:
>On 21 October 2005 21:12, Jacques Carette wrote:
>>I would strongly recommend against 'allow everyone to just commit'
>>without the presence of a large automated test suite which is used to
>>(automatically) reject code that breaks a test.
>pre-commit testing using GHC isn't really practical; it would be a huge
>bottleneck. The asynchronous testing we have right now (full testsuite
>every night) is a better compromise.
And for check-ins of changesets that affect a *lot* of tests, this is
also the compromise that was chosen at Maplesoft.
The upshot of that is that it
1) complicated the automated rejection of code a lot, enough so that
2) it necessitates having someone monitor nightly test failures every
day to manually figure out what managed to make things fail
#2 can vary from very little work to quite onerous, depending on the
level of activity. It only gets bad when things are allowed to
degenerate. Interestingly cases of many tests failures seem to be more
people-related than anything else. So a social process (ie making them
feel really guilty by letting everyone know who screwed up the build,
again) seems to reach a workable middle ground quite efficiently.
>However, online testing of libraries using Hugs might be do-able. It
>would make a nice little project if anyone's interested. I would
>happily run a bunch of Hugs tests before committing a library change if
>it was done either with a single command or automatically by the
I can't *do* this, but I can certainly help with the design. I helped
write parts of Maplesoft's "rfindtest" (script to find which tests to
run given a changeset) and "robocop" (script to find which changeset
most likely is the source of a particular test failure). The hardest
part is to ensure that one has the right data available in the right
format, the rest is easy database query stuff coupled with lots of
scripting. Being able to run the appropriate 100 tests (out of several
million) for a small change certainly is very convenient!
More information about the Libraries