Test suite quality

[Simon Marlow]

On 07/01/2012 17:52, Joachim Breitner wrote:
> Dear GHC-Team,
>
> to improve the quality of Haskell on Debian started to package the GHC
> testsuite in a package of its own, and will upload it to Debian so that
> it runs on all architectures and catches possible architecture-dependent
> bugs there.
>
> Using the released testsuite for 7.0.4 with that version of the
> compiler, and running it with
> 	 SKIPWAY="optc profc optllvm" fast=YES
> I get this result:
>
> OVERALL SUMMARY for test run started at Sa 7. Jan 17:45:54 CET 2012
>      2612 total tests, which gave rise to
>     10986 test cases, of which
>         0 caused framework failures
>      8642 were skipped
>
>      2267 expected passes
>        59 expected failures
>         0 unexpected passes
>        18 unexpected failures
>
> Unexpected failures:
>     3586(normal)
>     Chan001(normal)
>     MVar001(normal)
>     QSem001(normal)
>     QSemN001(normal)
>     SampleVar001(normal)
>     T1969(normal)
>     T3064(normal)
>     T3087(normal)
>     T3294(normal)
>     T4316(ghci)
>     T4801(normal)
>     cgrun025(normal)
>     driver027(normal)
>     driver028(normal)
>     ghci014(ghci)
>     ghcpkg05(normal)
>     regex003(normal)
>
> Is that normal to find unexpected failures of the release testsuite on a
> “normal” system?

Unfortunately yes, but we are getting better in this area.  Here's where 
we stand on the 7.4 branch for our major platforms currently:

  x86_64/Linux:  1 unexpected failure
  x86/Linux:     33 unexpected failures
  x86/Windows:   65 unexpected failures
  x86_64/Mac:    unknown (no nightly build?)

In fact the majority of these failures are just bugs in the tests or the 
test suite framework, and keeping the testsuite clean takes quite a lot 
of effort, which is why it's hard to get motivated to do it.  But I 
agree it's important, especially for a release, to have zero failures.

So I've made a ticket to do this for 7.4.1:

   http://hackage.haskell.org/trac/ghc/ticket/5757

> Should I file bug reports about unexpected failures found in the latest
> testsuite, once I get around to package and run that?

Sure, please do.

> And a minor request, in case someone feels like touching the test
> driver: It would be nice if the summary lists all library packages that
> are not installed but required by some test, i.e. the libraries that
> ought to be added to the Build-Dependency of the test suite to get a
> more complete coverage.

Yes, good idea.  I recently changed the driver so that it now prints out 
the number of failures due to missing dependencies, but it doesn't print 
out which packages are required, that would be a good improvement.

Cheers,
	Simon



> I could not attach the build log, as it is too large, but you can find
> it here:
> http://people.debian.org/~nomeata/ghc-testsuite_7.0.4-1_amd64.build.gz
>
> Thanks,
> Joachim
>
>
>
>
> _______________________________________________
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users at haskell.org
> http://www.haskell.org/mailman/listinfo/glasgow-haskell-users