[Haskell-cafe] Re: GSoC: Improving Cabal's Test Support

[Gregory Crosswhite]

Rather that starting from scratch, you should strongly consider adapting something like test-framework to this task, as it already has done the heavy work of creating a way to combine tests from different frameworks into a single suite and includes such features as displaying a progress bar during the QuickCheck tests.  Furthermore, it is easily extendable to support new kinds of tests;  for example, I found that it was relatively straightforward to add a new kind of "statistical" test to make sure that the average value of a function where where it should be.

Cheers,
Greg

On Apr 6, 2010, at 3:51 PM, Thomas Tuegel wrote:

> Hello again!
> 
> Based on the invaluable feedback I've received, I've made some
> revisions to the proposal I made a few days ago (at the end of this
> post, after my signature).  I apologize for the length of my post, but
> I'd like once again to solicit feedback on this.  Any commentary is
> very helpful!
> 
> Thanks!
> -- 
> Thomas Tuegel
> 
> 
> Throughout this proposal, examples are given to indicate how a package
> author would utilize the features proposed here.  In all these
> examples, suppose that the programmer is the author of the
> 'haskell-foo' package, which exposes the module 'Foo' and has a single
> test executable, 'foo-tests', using the QuickCheck testing library.
> 
> 
>    Package Description File Syntax
> 
> The syntax for designating test executables in package description
> files will be based on the existing syntax for describing executables.
> Such a stanza in the hypothetical package's description file would
> look like:
> 
>> Test foo-tests
>>    main-is: foo-tests.hs
>>    build-depends: haskell-foo, Cabal, QuickCheck
> 
> This example is obviously minimal; this is really an 'Executable'
> stanza by another name, so any options recognized there would also be
> valid here.
> 
> 
>    Handling of Test Executables by Cabal
> 
> The changes proposed here will make it possible to build, test, and
> install a Cabal package with the usual sequence of commands:
> 
> $ cabal configure
> $ cabal build
> $ cabal test
> $ cabal install
> 
> Cabal will recognize two new options during the 'configure' stage:
> '--enable-tests' and '--disable-tests'.
> 
> If 'cabal configure' is invoked with the '--enable-tests' option, then
> any test executables designated in the package description file will
> be built.  For the purposes of the 'configure' and 'build' stages,
> they will be handled as if they were ordinary executables, i.e.,
> described by 'Executable' stanzas.  With tests enabled, the test
> programs will be executed and their results collected by Cabal during
> the 'test' stage.
> 
> If 'cabal configure' is invoked with the '--disable-tests' option
> (which should be the default if neither option is specified), then
> test executables designated in the package description file will be
> ignored, as if the 'Test' stanza were absent.  Any attempt to invoke
> the 'test' stage with tests disabled should remind the user of that
> fact.
> 
> Regardless of the status of tests (enabled or disabled), the 'install'
> stage will ignore any executables designated as test suites, since it
> is not desirable to install the test executables.
> 
> 
>    Collection of Test Results
> 
> Cabal will provide a standard interface, residing in the module
> 'Distribution.Test', for running tests independent of the testing
> library used.  A minimal outline of this module looks like:
> 
>> module Distribution.Test where
>> 
>> type Name = String
>> type Result = Maybe Bool
>> type Info = String
>> type Output = String
>> 
>> -- 'Compiler' and 'ComponentLocalBuildInfo' are already provided by Cabal.
>> -- They are included here to aid in debugging test failures
>> type Report = (Compiler, ComponentLocalBuildInfo, [(Name, Result, Info, Output)])
>> 
>> class Test t where
>>    wrap :: t -> IO (Result, Info)
>> 
>> runTests :: Test t => [(Name, t)] -> IO Report
>> 
>> writeResults :: Report -> IO ()
> 
> Instances of 'Test' will run a type of test from one of the testing
> libraries; part of this project will therefore be patching QuickCheck
> and HUnit to provide these instances.  Any other testing library
> providing this instance will also be compatible with the automated
> testing features this proposal introduces.
> 
> The type 'Maybe Bool' is used throughout this framework to indicate a
> test result: 'Nothing' indicates a test was not run, 'Just False'
> indicates a failed test, and 'Just True' indicates a successful test.
> The 'Info' string captures any information provided by the testing
> library.  However, because of the reliance of most test suites on
> standard output, Cabal will also capture the standard output produced
> during each test (when the test suite is invoked through 'cabal
> test'); the output will be included in the test result file.
> 
> The function 'writeResults' will write the test results to a file.
> The 'Show' instance for the type of its single argument will therefore
> constitute the standard test result file format.  This has the
> advantage of being human- and machine-readable without requiring any
> extra dependencies to parse the file.
> 
> With this framework, the hypothetical package's author might write a
> test suite such as:
> 
>> module Main where
>> 
>> import Distribution.Test
>> import Foo
>> import QuickCheck
>> 
>> testBar :: Gen Bool
>> testBar = ...
>> 
>> testBaz :: Gen Bool
>> testBaz = ...
>> 
>> main = runTests [("testBar", testBar), ("testBaz", testBaz)] >>= writeResults
> 
> 
>    Reporting and Comparing Test Results
> 
> The 'cabal test' command will run tests by default, but support two
> other options:
> 
>    1.  '--report [file]', which will produce a nicely formatted
> report of the test results stored in the named file, or of the last
> run of the package's test suite if no file is specified, and
>    2.  '--diff file1 file2', which will show the differences between
> the test results stored it two different files.
> 
> Because the report file format is readily parseable by any Haskell
> program, it could be processed into another format for compatibility
> with existing tools.
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://www.haskell.org/mailman/listinfo/haskell-cafe