[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Gregory Crosswhite]

If the goal is continuous integration, perhaps it would be sufficient to require "cabal test" to return an error code of 0 if all tests succeed, and something else if any of them fail;  it can additionally print whatever output it wants in either case.  The continuous integration system would then run "cabal test" after the build, and if it succeeded (error code 0) say nothing, and if it failed (error code something else) it would report that the build failed and show the output from "cabal test" to give details to the developer.

Cheers,
Greg

On Apr 28, 2010, at 12:55 PM, Rogan Creswick wrote:

> On Wed, Apr 28, 2010 at 8:19 AM, Duncan Coutts
> <duncan.coutts at googlemail.com> wrote:
>> 
>> Yes, it means the testing agent (cabal-install or some other
>> program/system) can do more than simply run all the tests. It means it
>> can enumerate them and not run them (think a GUI or web interface), run
>> a subset of tests, run them in parallel etc.
>> 
> 
> I'm not convinced that this should be cabal's responsibility.
> 
> I think we would be better served by leaving this up to the test
> frameworks (indeed, test-framework has test filtering capabilities
> already).  If 'cabal test' simply acts as a thin layer between the
> user/invoking system and the test framework, then we could pass
> arguments through to the underlying test binary and perform these
> tasks using whatever interface that test binary provides.  This will
> buy us more flexibility in the long run.  (I think this is at least a
> good place to start -- and matches my interpretation of Thomas's
> proposal.)
> 
> If Cabal takes on these responsibilities, then the testing api will be
> more constrained -- we won't be able to experiment with new test
> formats/methodologies as easily, since any tests will have to meet a
> specific API.
> 
> While I agree that we need standardization, I think that we should
> achieve that by using compatible output formats and compatible (user)
> interfaces (and enforcing those with tests, schema checkers, etc..).
> I don't see many benefits to baking this functionality into cabal when
> it could be done separately.
> 
> --Rogan
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe