[Haskell-cafe] GSoC: Improving Cabal's Test Support

[Thomas Tuegel]

On Thu, Apr 1, 2010 at 9:36 PM, Ivan Lazar Miljenovic
<ivan.miljenovic at gmail.com> wrote:
>> I propose to build a test suite as its own executable, but to avoid
>> the problem of granularity by producing an output file detailing the
>> success or failure of individual tests and any relevant error
>> messages.  The format of the file would be standardized through
>> library routines I propose to write; these routines would run tests
>> with HUnit or QuickCheck and process them into a common format.
>> Cabal, or any other utility, could read this file to determine the
>> state of the test suite.  Perhaps Cabal could even warn the user about
>> installing packages with failing tests.
>
> All well and good, but your sample code (which I've ommitted for the
> sake of brevity) doesn't seem to lead to much room for customisation:
> for graphviz's test suite, I include various comments about the purpose
> of the test, etc. as well as changing some of QuickCheck's paramaters;
> does your proposal allow the inclusion of such customisations?

I've been looking at graphviz's test suite.  You used a type

> data Test = Test { name :: String
>                  , desc :: String
>                  , test :: IO Result -- ^ QuickCheck test.
>                  }

to represent tests (sorry for mangling the indentation), but this is
practically the same as the type I suggested:

> type Test = (String, IO (Bool, String))

The String output in the IO monad can just as easily be used to output
a test description for a successful test as it can be used for an
error message for a failing test.  Other than that, the only
difference is that you keep the entire QuickCheck Result, and I
proposed just to keep a Bool (for compatibility between QuickCheck and
HUnit).  The use of Bool is purely illustrative; if I use one of the
frameworks I've been discussing with Rogan, it will be whatever that
framework uses, instead of a Bool.

> As an aside, I question the necessity of this kind of proposal: how many
> people actually run tests for packages they download from Hackage?
> graphviz's test suite runs for 110 minutes, and I mainly use it to test
> that my changes/inclusions in what it prints and parses is consistent: I
> don't expect a user to bother running it (and would question why anyone
> would).

It's true that this proposal does much more for package authors than
for users.  I'm not proposing to require users to run tests for every
package.  At the same time, I'm sure that on the Linux side,
distribution packagers would appreciate wider and more uniform
availability of package tests; this proposal would help on both
fronts.

> How does the inclusion of a "test" option to Cabal allow any
> substantial benefits to developers over building the package with a
> build-time flag to enable building a test suite?

Admittedly, you stand to benefit less than a package author without
such a complete test suite.  At the very least, the proposed 'Test'
stanza for the package description file would make it unnecessary for
you to worry about recognizing a build-time flag and messing about
with UserHooks.  (As I said to Rogan, using the 'Test' stanza wouldn't
lock you in to a particular framework: you could use it to compile
your test suite as-is.)  That lowers the barrier to entry for creating
test suites so more authors will use them, and that's a substantial
benefit.

I also envision automated testing being offered as a service for
package authors: I would like, e.g., to ensure that my package works
correctly on different architectures and platforms, but I don't have
the means to acquire all those different machines.  Collectively,
however, the community does.  If we used a standard format for
handling test results, we could offer a service where uploaded
packages would be compiled and tested on an array of machines.  Since
we're using a common format, we can write one tool to parse and
summarize the results across all those platforms, and authors can
immediately see where their packages are failing.  Certainly, any
single package author could write a tool to do this for any single
package's tests, but now we have a common tool everyone can use simply
by writing tests.

To summarize: everyone's life is easier if we do this bit of plumbing for them.

-- 
Thomas Tuegel