Test suite quality
marlowsd at gmail.com
Mon Jan 9 12:36:10 CET 2012
On 07/01/2012 17:52, Joachim Breitner wrote:
> Dear GHC-Team,
> to improve the quality of Haskell on Debian started to package the GHC
> testsuite in a package of its own, and will upload it to Debian so that
> it runs on all architectures and catches possible architecture-dependent
> bugs there.
> Using the released testsuite for 7.0.4 with that version of the
> compiler, and running it with
> SKIPWAY="optc profc optllvm" fast=YES
> I get this result:
> OVERALL SUMMARY for test run started at Sa 7. Jan 17:45:54 CET 2012
> 2612 total tests, which gave rise to
> 10986 test cases, of which
> 0 caused framework failures
> 8642 were skipped
> 2267 expected passes
> 59 expected failures
> 0 unexpected passes
> 18 unexpected failures
> Unexpected failures:
> Is that normal to find unexpected failures of the release testsuite on a
> “normal” system?
Unfortunately yes, but we are getting better in this area. Here's where
we stand on the 7.4 branch for our major platforms currently:
x86_64/Linux: 1 unexpected failure
x86/Linux: 33 unexpected failures
x86/Windows: 65 unexpected failures
x86_64/Mac: unknown (no nightly build?)
In fact the majority of these failures are just bugs in the tests or the
test suite framework, and keeping the testsuite clean takes quite a lot
of effort, which is why it's hard to get motivated to do it. But I
agree it's important, especially for a release, to have zero failures.
So I've made a ticket to do this for 7.4.1:
> Should I file bug reports about unexpected failures found in the latest
> testsuite, once I get around to package and run that?
Sure, please do.
> And a minor request, in case someone feels like touching the test
> driver: It would be nice if the summary lists all library packages that
> are not installed but required by some test, i.e. the libraries that
> ought to be added to the Build-Dependency of the test suite to get a
> more complete coverage.
Yes, good idea. I recently changed the driver so that it now prints out
the number of failures due to missing dependencies, but it doesn't print
out which packages are required, that would be a good improvement.
> I could not attach the build log, as it is too large, but you can find
> it here:
> Glasgow-haskell-users mailing list
> Glasgow-haskell-users at haskell.org
More information about the Glasgow-haskell-users