Controlled anarchy, was Re: Data.* collections maintenance

[Jacques Carette]

Simon Marlow wrote:

>On 21 October 2005 21:12, Jacques Carette wrote:
>  
>
>>I would strongly recommend against 'allow everyone to just commit'
>>without the presence of a large automated test suite which is used to
>>(automatically) reject code that breaks a test.
>>    
>>
>
>pre-commit testing using GHC isn't really practical; it would be a huge
>bottleneck.  The asynchronous testing we have right now (full testsuite
>every night) is a better compromise.
>  
>
And for check-ins of changesets that affect a *lot* of tests, this is 
also the compromise that was chosen at Maplesoft.

The upshot of that is that it
1) complicated the automated rejection of code a lot, enough so that
2) it necessitates having someone monitor nightly test failures every 
day to manually figure out what managed to make things fail

#2 can vary from very little work to quite onerous, depending on the 
level of activity.  It only gets bad when things are allowed to 
degenerate.  Interestingly cases of many tests failures seem to be more 
people-related than anything else.  So a social process (ie making them 
feel really guilty by letting everyone know who screwed up the build, 
again) seems to reach a workable middle ground quite efficiently.

>However, online testing of libraries using Hugs might be do-able.  It
>would make a nice little project if anyone's interested.  I would
>happily run a bunch of Hugs tests before committing a library change if
>it was done either with a single command or automatically by the
>checkin.
>
I can't *do* this, but I can certainly help with the design.  I helped 
write parts of Maplesoft's "rfindtest" (script to find which tests to 
run given a changeset) and "robocop" (script to find which changeset 
most likely is the source of a particular test failure).  The hardest 
part is to ensure that one has the right data available in the right 
format, the rest is easy database query stuff coupled with lots of 
scripting.  Being able to run the appropriate 100 tests (out of several 
million) for a small change certainly is very convenient!

Jacques