[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Duncan Coutts]

On Wed, 2010-04-28 at 09:55 -0700, Rogan Creswick wrote:
> On Wed, Apr 28, 2010 at 8:19 AM, Duncan Coutts
> <duncan.coutts at googlemail.com> wrote:
> >
> > Yes, it means the testing agent (cabal-install or some other
> > program/system) can do more than simply run all the tests. It means it
> > can enumerate them and not run them (think a GUI or web interface), run
> > a subset of tests, run them in parallel etc.
> >
> 
> I'm not convinced that this should be cabal's responsibility.

"Cabal" should define the interface between testsuite and test runner.
Nothing more.

Packages should provide collections of tests.

Testing agents should provide a test runner to actually run the tests
and do something with the results.

See for example test-framework which has exactly this decomposition
between testsuites (a collection of tests) and a test runner. They are
mediated by a common interface. The test-framework package provides both
the interface, some adapters for QC/HUnit to provide tests, and also a
sample test runner that prints results to the console.

Tools like cabal-install that use the interface defined by Cabal can
provide a test runner (almost certainly implemented in some other
package) and then do something interesting with the results like showing
them to the user or uploading them to hackage.

Other tools can use other test runners and do other interesting things.

> I think we would be better served by leaving this up to the test
> frameworks (indeed, test-framework has test filtering capabilities
> already).  If 'cabal test' simply acts as a thin layer between the
> user/invoking system and the test framework, then we could pass
> arguments through to the underlying test binary and perform these
> tasks using whatever interface that test binary provides.  This will
> buy us more flexibility in the long run.  (I think this is at least a
> good place to start -- and matches my interpretation of Thomas's
> proposal.)
> 
> If Cabal takes on these responsibilities, then the testing api will be
> more constrained -- we won't be able to experiment with new test
> formats/methodologies as easily, since any tests will have to meet a
> specific API.

It is exactly defining this API that should allow flexibility. It means
packages can provide tests and have them be used in multiple different
ways by different test runners with different purposes and capabilities.

If all you can do is run the testsuite and collect the results, that's
much more constrained.

Note that we're also proposing a lowest-common-denominator testsuite
interface that gives all the control to the package author, but means
the test runner cannot interpret the results in any interesting way. The
main point is to wrap any existing testsuite in a way that allows it to
be run automatically and non-iteractively for e.g. hackage testing.

Duncan