Test suite quality

[Joachim Breitner]

Dear GHC-Team,

to improve the quality of Haskell on Debian started to package the GHC
testsuite in a package of its own, and will upload it to Debian so that
it runs on all architectures and catches possible architecture-dependent
bugs there.

Using the released testsuite for 7.0.4 with that version of the
compiler, and running it with
	 SKIPWAY="optc profc optllvm" fast=YES
I get this result:

OVERALL SUMMARY for test run started at Sa 7. Jan 17:45:54 CET 2012
    2612 total tests, which gave rise to
   10986 test cases, of which
       0 caused framework failures
    8642 were skipped

    2267 expected passes
      59 expected failures
       0 unexpected passes
      18 unexpected failures

Unexpected failures:
   3586(normal)
   Chan001(normal)
   MVar001(normal)
   QSem001(normal)
   QSemN001(normal)
   SampleVar001(normal)
   T1969(normal)
   T3064(normal)
   T3087(normal)
   T3294(normal)
   T4316(ghci)
   T4801(normal)
   cgrun025(normal)
   driver027(normal)
   driver028(normal)
   ghci014(ghci)
   ghcpkg05(normal)
   regex003(normal)

Is that normal to find unexpected failures of the release testsuite on a
“normal” system? Would it be reasonable to have the released testsuite,
at least in fast mode, complete without unexpected failures so that I
can be sure that any unexpected failure implies some problem in the
compiler?

Should I file bug reports about unexpected failures found in the latest
testsuite, once I get around to package and run that?

And a minor request, in case someone feels like touching the test
driver: It would be nice if the summary lists all library packages that
are not installed but required by some test, i.e. the libraries that
ought to be added to the Build-Dependency of the test suite to get a
more complete coverage.

I could not attach the build log, as it is too large, but you can find
it here:
http://people.debian.org/~nomeata/ghc-testsuite_7.0.4-1_amd64.build.gz

Thanks,
Joachim

-- 
Joachim "nomeata" Breitner
Debian Developer
  nomeata at debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
  JID: nomeata at joachim-breitner.de | http://people.debian.org/~nomeata

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://www.haskell.org/pipermail/glasgow-haskell-users/attachments/20120107/8f45a075/attachment.pgp>