Unreliability of the build system

[Gabriel Dos Reis]

On Fri, May 17, 2013 at 3:14 PM, Ian Lynagh <ian at well-typed.com> wrote:
> On Fri, May 17, 2013 at 11:34:01AM -0500, Gabriel Dos Reis wrote:
>>
>> Can't the build bots instructed to systematically fire up on every
>> commit for testing on the various platforms you currently do?
>
> In principle, yes, although (a) many of them are on machines that are
> used to do something else during the day, and (b) the builders may often
> still be building the last patches when more patches are pushed.

For (a), we've benefited from the compile farm project made possible by
the Ada-Europe people:

    http://gcc.gnu.org/wiki/CompileFarm

It supports open source projects, and I don't think it is
restricted only to GCC people.   It is available to people
developing open source project (see the webpage for
details.) Maybe the Haskell developer community might
want to give it a try or set up similar projects.
This is a community thing, and it takes a community to grow.

For (b) you may consider job queues and/or concurrent compiles.

> We already get testsuite results mailed to the list each night, though.

Yes, I see those.  But from your previous description it sounds
to me as though they would catch some particular cluster of commits
that may cumulatively break things with no one taking responsibility
or stepping forward to fix (because they didn't break it in the first place.)

> What we need is for people to fix unexpected results in them.

Agreed.  But, from my 15+ years with GCC, it makes a huge
difference when you have an automated assistance that can pinpoint
breakages to specific commits/offenders.

Another thing I've found useful was the "24 hours" rule whereby
when a breakage was identified to be caused by a specific
commits, the author is given 24 hours to fix it.  Beyond that, if
no action is taken by the committer and/or if it is clear that it
will take much longer to fix then a reversal patch is pre-approved.
That helps getting people clean their mess :-)

> (admittedly we don't currently identify who/which patches caused a test
> to start failing; perhaps that would help).

Yes, exactly!

-- Gaby