Unreliability of the build system

[Jan Stolarek]

Fellow developers,

I need advice on using GHC build system. Recently I've been spending more time on resolving 
problems with the build system itself than on developing GHC, and this is not how things are 
supposed to be. I have found testsuite and validation to be very unreliable - things seem to 
break randomly and results are not repeatable. I suspect that I might be doing something wrong 
and I will appreciate any advice or explanations why I'm having these problem.

I have three GHC trees:
  * ghc-working  - source tree I use for development
  * ghc-build    - lndir'ed to ghc-working, used for building
  * ghc-validate - source tree used for validation, ghc-working is set up as one of the remotes

Here's the situation from yesterday evening. I updated ghc-working and ghc-validate trees from the 
origin - according to the fingerprint.py script both trees were identical. I ran validation in 
ghc-validate and ghc-build trees (on the master branches, no my changes). The only difference 
between them should be that ghc-build is symlinked. Validation in ghc-validate passed with two 
failures (others on IRC confirmed they are having same failures), while validation in ghc-build 
failed with:

Haddock coverage:
 100% (  1 /  1) in 'GHC.IntWord64'
  78% (  7 /  9) in 'GHC.Types'
  17% (  1 /  6) in 'GHC.CString'
   3% (  2 / 63) in 'GHC.Tuple'
   0% (  0 /  3) in 'GHC.Debug'
   0% (  0 /342) in 'GHC.PrimopWrappers'
  30% (160 /531) in 'GHC.Prim'
 100% (  3 /  3) in 'GHC.Magic'
  38% (  6 / 16) in 'GHC.Classes'
haddock: internal error: /dane/uczelnia/projekty/ghc-build/inpl
ace/lib/html/Ocean.std-theme/ocean.css: copyFile: does not exis
t (No such file or directory)
gmake[1]: *** [libraries/ghc-prim/dist-install/doc/html/ghc-pri
m/ghc-prim.haddock] Błąd 1
gmake[1]: *** Kasuję plik `libraries/ghc-prim/dist-install/doc/
html/ghc-prim/ghc-prim.haddock' 

Today I ran ./sync-all pull in both trees, but no new commits were added (according to 
fingerprint). I run the validation again and the validation in ghc-build fails as previously, 
while the validation in ghc-validate reports over 200 unexpected failures. What is more, I found 
that rerunning some of the failing tests (make TEST=whatever) often results with these tests 
passing. This unreliability is very frustrating.

I've been having these problems since I started working on GHC in December. I'm wondering whether 
this is a common thing or am I doing something not-so-obviously wrong? As usual, I will 
appreciate any advice from those more experienced than me.

Janek