[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Thomas Tuegel]

Hello again!

Based on the invaluable feedback I've received, I've made some
revisions to the proposal I made a few days ago (at the end of this
post, after my signature).  I apologize for the length of my post, but
I'd like once again to solicit feedback on this.  Any commentary is
very helpful!

Thanks!
-- 
Thomas Tuegel


Throughout this proposal, examples are given to indicate how a package
author would utilize the features proposed here.  In all these
examples, suppose that the programmer is the author of the
'haskell-foo' package, which exposes the module 'Foo' and has a single
test executable, 'foo-tests', using the QuickCheck testing library.


    Package Description File Syntax

The syntax for designating test executables in package description
files will be based on the existing syntax for describing executables.
 Such a stanza in the hypothetical package's description file would
look like:

> Test foo-tests
>     main-is: foo-tests.hs
>     build-depends: haskell-foo, Cabal, QuickCheck

This example is obviously minimal; this is really an 'Executable'
stanza by another name, so any options recognized there would also be
valid here.


    Handling of Test Executables by Cabal

The changes proposed here will make it possible to build, test, and
install a Cabal package with the usual sequence of commands:

$ cabal configure
$ cabal build
$ cabal test
$ cabal install

Cabal will recognize two new options during the 'configure' stage:
'--enable-tests' and '--disable-tests'.

If 'cabal configure' is invoked with the '--enable-tests' option, then
any test executables designated in the package description file will
be built.  For the purposes of the 'configure' and 'build' stages,
they will be handled as if they were ordinary executables, i.e.,
described by 'Executable' stanzas.  With tests enabled, the test
programs will be executed and their results collected by Cabal during
the 'test' stage.

If 'cabal configure' is invoked with the '--disable-tests' option
(which should be the default if neither option is specified), then
test executables designated in the package description file will be
ignored, as if the 'Test' stanza were absent.  Any attempt to invoke
the 'test' stage with tests disabled should remind the user of that
fact.

Regardless of the status of tests (enabled or disabled), the 'install'
stage will ignore any executables designated as test suites, since it
is not desirable to install the test executables.


    Collection of Test Results

Cabal will provide a standard interface, residing in the module
'Distribution.Test', for running tests independent of the testing
library used.  A minimal outline of this module looks like:

> module Distribution.Test where
>
> type Name = String
> type Result = Maybe Bool
> type Info = String
> type Output = String
>
> -- 'Compiler' and 'ComponentLocalBuildInfo' are already provided by Cabal.
> -- They are included here to aid in debugging test failures
> type Report = (Compiler, ComponentLocalBuildInfo, [(Name, Result, Info, Output)])
>
> class Test t where
>     wrap :: t -> IO (Result, Info)
>
> runTests :: Test t => [(Name, t)] -> IO Report
>
> writeResults :: Report -> IO ()

Instances of 'Test' will run a type of test from one of the testing
libraries; part of this project will therefore be patching QuickCheck
and HUnit to provide these instances.  Any other testing library
providing this instance will also be compatible with the automated
testing features this proposal introduces.

The type 'Maybe Bool' is used throughout this framework to indicate a
test result: 'Nothing' indicates a test was not run, 'Just False'
indicates a failed test, and 'Just True' indicates a successful test.
The 'Info' string captures any information provided by the testing
library.  However, because of the reliance of most test suites on
standard output, Cabal will also capture the standard output produced
during each test (when the test suite is invoked through 'cabal
test'); the output will be included in the test result file.

The function 'writeResults' will write the test results to a file.
The 'Show' instance for the type of its single argument will therefore
constitute the standard test result file format.  This has the
advantage of being human- and machine-readable without requiring any
extra dependencies to parse the file.

With this framework, the hypothetical package's author might write a
test suite such as:

> module Main where
>
> import Distribution.Test
> import Foo
> import QuickCheck
>
> testBar :: Gen Bool
> testBar = ...
>
> testBaz :: Gen Bool
> testBaz = ...
>
> main = runTests [("testBar", testBar), ("testBaz", testBaz)] >>= writeResults


    Reporting and Comparing Test Results

The 'cabal test' command will run tests by default, but support two
other options:

    1.  '--report [file]', which will produce a nicely formatted
report of the test results stored in the named file, or of the last
run of the package's test suite if no file is specified, and
    2.  '--diff file1 file2', which will show the differences between
the test results stored it two different files.

Because the report file format is readily parseable by any Haskell
program, it could be processed into another format for compatibility
with existing tools.