[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

Duncan Coutts duncan.coutts at googlemail.com
Thu Apr 8 08:53:39 EDT 2010


On Wed, 2010-04-07 at 16:09 -0400, Thomas Tuegel wrote:
> On Wed, Apr 7, 2010 at 3:33 PM, Duncan Coutts
> <duncan.coutts at googlemail.com> wrote:
> > The importance of this is that it lets us develop improved testsuite
> > interfaces in future. At the moment there are two test interfaces we
> > want to support. One is the simple unix style "exit code + stdout"
> > interface. This is good because it is a lowest common denominator that
> > all existing testsuites can fit into.
> >
> > Of course that test interface does not provide any detailed
> > machine-readable information (though you do get human-readable test
> > logs). So that's why we want a second interface. That one should let the
> > "testing agent" (for example "cabal test" but could be other agents) get
> > much more detail about what tests can be run and then what the results
> > are of various tests.
> 
> For the purpose of differentiating between these two, would a field in
> the test section such as "interface: stdout" (in the first case) or
> "interface: detailed" (in the second) suffice?

Yep.

> > The details of such an interface are up for discussion. I do not mind if
> > that is a command line executable interface or a library interface.
> 
> That's something I've been thinking about.  The former seems more
> portable.  Maybe cabal could call the test program with "test_program
> --list" to produce a list of tests and "test_program
> test1,test2,test3" to run some tests.

Having some ideas is good. The details of the interface can be worked
out during the project.

> I also want to ask how strictly this is within the scope of the SoC
> project, i.e. how well will my proposal be received if it focuses
> primarily on the first part of the problem (getting everything working
> for the stdout interface)?  I ask because the detailed interface seems
> like a much larger mandate given that cabal doesn't really support any
> of the syntax/features for the simple stdout interface.

Certainly we want to get the first part working first. My guess is that
there is time within the 3-month GSoC period to complete the basic bits
and to at least have a go at a prototype of a more detailed interface.
That part doesn't need to be final, especially given that it will be
possible to create new interfaces in future.

> >>     Handling of Test Executables by Cabal
> >>
> >> The changes proposed here will make it possible to build, test, and
> >> install a Cabal package with the usual sequence of commands:
> >
> > That all sounds reasonable. I'd like to add that the interface between
> > the testsuite and a testing agent such as the cabal program should be
> > clearly documented and specified. There will likely be dedicated test
> > agents that want to run the tests too and send reports to other systems
> > (e.g. dedicated hackage test agents) and convert to other formats (e.g.
> > integration in companies in-house systems).
> 
> Rogan mentioned possible upcoming support in test-framework for JUnit
> XML as an output format for test results.  That certainly seems to be
> widely supported; do you think it is suitable?

I think it's important to be able to convert into standard or custom
formats. I've no idea if JUnit XML would make sense as the native
format. It's plausible.

Duncan



More information about the Haskell-Cafe mailing list