[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Duncan Coutts]

On Tue, 2010-04-06 at 18:51 -0400, Thomas Tuegel wrote:
> Hello again!
> 
> Based on the invaluable feedback I've received, I've made some
> revisions to the proposal I made a few days ago (at the end of this
> post, after my signature).  I apologize for the length of my post, but
> I'd like once again to solicit feedback on this.  Any commentary is
> very helpful!

Hia Thomas.

>     Package Description File Syntax
> 
> The syntax for designating test executables in package description
> files will be based on the existing syntax for describing executables.
>  Such a stanza in the hypothetical package's description file would
> look like:
> 
> > Test foo-tests
> >     main-is: foo-tests.hs
> >     build-depends: haskell-foo, Cabal, QuickCheck

One feature that I consider to be vital (and as Cabal maintainer I get
to say that kind of thing! ;-) ) is that the stanza must specify the
"testing interface" that it supports.

The importance of this is that it lets us develop improved testsuite
interfaces in future. At the moment there are two test interfaces we
want to support. One is the simple unix style "exit code + stdout"
interface. This is good because it is a lowest common denominator that
all existing testsuites can fit into.

Of course that test interface does not provide any detailed
machine-readable information (though you do get human-readable test
logs). So that's why we want a second interface. That one should let the
"testing agent" (for example "cabal test" but could be other agents) get
much more detail about what tests can be run and then what the results
are of various tests.

The details of such an interface are up for discussion. I do not mind if
that is a command line executable interface or a library interface.

>     Handling of Test Executables by Cabal
> 
> The changes proposed here will make it possible to build, test, and
> install a Cabal package with the usual sequence of commands:

That all sounds reasonable. I'd like to add that the interface between
the testsuite and a testing agent such as the cabal program should be
clearly documented and specified. There will likely be dedicated test
agents that want to run the tests too and send reports to other systems
(e.g. dedicated hackage test agents) and convert to other formats (e.g.
integration in companies in-house systems).

A "cabal test" user interface is obviously great for developers.

Gregory makes a goof suggestion about using or adapting the existing
test-framework package. That was also something I was thinking about. It
would be good to work with the maintainer of the test-framework package
so that it can be used to implement the interface that Cabal specifies.

Duncan