[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Thomas Tuegel]

On Wed, Apr 28, 2010 at 12:55 PM, Rogan Creswick <creswick at gmail.com> wrote:
> On Wed, Apr 28, 2010 at 8:19 AM, Duncan Coutts
> <duncan.coutts at googlemail.com> wrote:
>>
>> Yes, it means the testing agent (cabal-install or some other
>> program/system) can do more than simply run all the tests. It means it
>> can enumerate them and not run them (think a GUI or web interface), run
>> a subset of tests, run them in parallel etc.
>>
>
> I'm not convinced that this should be cabal's responsibility.
>
> I think we would be better served by leaving this up to the test
> frameworks (indeed, test-framework has test filtering capabilities
> already).  If 'cabal test' simply acts as a thin layer between the
> user/invoking system and the test framework, then we could pass
> arguments through to the underlying test binary and perform these
> tasks using whatever interface that test binary provides.  This will
> buy us more flexibility in the long run.  (I think this is at least a
> good place to start -- and matches my interpretation of Thomas's
> proposal.)

That is more or less how I intended my proposal to be read, the caveat
being that I intentionally said very little about the detailed test
suite interface.  I agree that we should leave much up to the testing
frameworks.

If we start implementing facilities in Cabal to pick and choose
specific tests from inside test suites, we're essentially writing yet
another test framework into Cabal; I've been specifically discouraged
from doing that since I began discussing this proposal on the list.  I
am increasingly of the opinion that we should just provide a simple,
stdout/exit code interface and let test frameworks handle the rest:

If developers want continuous integration with existing testing tools,
the can use a framework that supports the output format those tools
use, and pipe that output to stdout to be captured by Cabal.  Then
they can turn whichever tool they want loose on the output file.

If developers want to independently run subsets of their tests, they
can give them independent test stanzas in the .cabal file.  Either
they put the tests in different executables, or the test framework can
provide command-line options for turning tests on and off.

Those are the big two usage scenarios we've discussed for the detailed
test interface, and I think these examples demonstrate why I think it
may be unnecessary.

-- 
Thomas Tuegel