[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Thomas Tuegel]

On Wed, Apr 7, 2010 at 3:33 PM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:
> The importance of this is that it lets us develop improved testsuite
> interfaces in future. At the moment there are two test interfaces we
> want to support. One is the simple unix style "exit code + stdout"
> interface. This is good because it is a lowest common denominator that
> all existing testsuites can fit into.
>
> Of course that test interface does not provide any detailed
> machine-readable information (though you do get human-readable test
> logs). So that's why we want a second interface. That one should let the
> "testing agent" (for example "cabal test" but could be other agents) get
> much more detail about what tests can be run and then what the results
> are of various tests.

For the purpose of differentiating between these two, would a field in
the test section such as "interface: stdout" (in the first case) or
"interface: detailed" (in the second) suffice?

> The details of such an interface are up for discussion. I do not mind if
> that is a command line executable interface or a library interface.

That's something I've been thinking about.  The former seems more
portable.  Maybe cabal could call the test program with "test_program
--list" to produce a list of tests and "test_program
test1,test2,test3" to run some tests.

I also want to ask how strictly this is within the scope of the SoC
project, i.e. how well will my proposal be received if it focuses
primarily on the first part of the problem (getting everything working
for the stdout interface)?  I ask because the detailed interface seems
like a much larger mandate given that cabal doesn't really support any
of the syntax/features for the simple stdout interface.

>>     Handling of Test Executables by Cabal
>>
>> The changes proposed here will make it possible to build, test, and
>> install a Cabal package with the usual sequence of commands:
>
> That all sounds reasonable. I'd like to add that the interface between
> the testsuite and a testing agent such as the cabal program should be
> clearly documented and specified. There will likely be dedicated test
> agents that want to run the tests too and send reports to other systems
> (e.g. dedicated hackage test agents) and convert to other formats (e.g.
> integration in companies in-house systems).

Rogan mentioned possible upcoming support in test-framework for JUnit
XML as an output format for test results.  That certainly seems to be
widely supported; do you think it is suitable?

-- 
Thomas Tuegel