[Haskell-cafe] GSoC: Improving Cabal's Test Support

[Ivan Lazar Miljenovic]

Thomas Tuegel <ttuegel at gmail.com> writes:
> There have been two separate suggestions (of which I am aware) of ways
> to integrate tests into Cabal.  One is to build the tests into their
> own executable which uses an error code on exit to indicate test
> failure.

I personally prefer this suggestion: for my graphviz library at least, I
define at least one QuickCheck property that I cannot as yet actually
use (as my Arbitrary instances aren't guaranteed to produce valid Dot
code, so I can't run the test that attempts to pass a value through to
dot/neato).

> I propose to build a test suite as its own executable, but to avoid
> the problem of granularity by producing an output file detailing the
> success or failure of individual tests and any relevant error
> messages.  The format of the file would be standardized through
> library routines I propose to write; these routines would run tests
> with HUnit or QuickCheck and process them into a common format.
> Cabal, or any other utility, could read this file to determine the
> state of the test suite.  Perhaps Cabal could even warn the user about
> installing packages with failing tests.

All well and good, but your sample code (which I've ommitted for the
sake of brevity) doesn't seem to lead to much room for customisation:
for graphviz's test suite, I include various comments about the purpose
of the test, etc. as well as changing some of QuickCheck's paramaters;
does your proposal allow the inclusion of such customisations?

As an aside, I question the necessity of this kind of proposal: how many
people actually run tests for packages they download from Hackage?
graphviz's test suite runs for 110 minutes, and I mainly use it to test
that my changes/inclusions in what it prints and parses is consistent: I
don't expect a user to bother running it (and would question why anyone
would).  How does the inclusion of a "test" option to Cabal allow any
substantial benefits to developers over building the package with a
build-time flag to enable building a test suite?

-- 
Ivan Lazar Miljenovic
Ivan.Miljenovic at gmail.com
IvanMiljenovic.wordpress.com