Next steps for "cabal test" feature

Max Bolingbroke batterseapower at hotmail.com
Mon Oct 11 18:13:36 EDT 2010


On 11 October 2010 13:38, Duncan Coutts <duncan.coutts at googlemail.com> wrote:
> But yes, a review at some point would be appreciated. The goal is to
> have something close to what test-framework has always had (ie tree of
> tests, QC, HUnit, other tests to be lifted in) but using as few language
> extensions as we can get away with. I also rather like Thomas'
> pure/impure distinction. It's also important of course to handle things
> like RNG seeds so that we can reproduce test runs.

OK, I've done a bit of a review.

Features of test-framework not in "cabal test":
 * Test groups
 * Ability to see summary of test results by test type (e.g. 2 failed
properties, 5 failed test cases)
 * Incremental generation of test results (so you can e.g. see
interactively the # of properties run so far while a property is still
executing)

Features of "cabal test" not in test-framework:
 * The result data type lets you distinguish between "errors" and "failures"
 * More extensible options type (i.e. it *is* actually extensible :-)

Questionable features of "cabal test":
 * I'm not too sure what the "options" member of "TestOptions" gives
you - after all, the test has
   to do it's own parsing of the [(String, String)] Options list, in
"check" and "run"/"runM" - so Cabal
   doesn't really get to use this information
 * Separation of pure/impure tests - what do you want to use this for?
Good unit tests should be isolated from each other (no dependencies).
Given this constraint, we can safely run IO-type unit tests in
parallel with each other and so on - so it seems that knowing they are
pure is not of great benefit?

We could take a union of these two feature sets to get the final
"cabal test" interface.

Alternatively we could take the union of the sets less the
"incremental generation" feature of test-framework. This is a rather
nice feature for those running tests interactively. However, it is
also one of the most technically challenging parts of the code - for a
flavour of the stuff it requires please see my "Improving IO" monad at
http://github.com/batterseapower/test-framework/blob/master/core/Test/Framework/Improving.hs.
The idea of this monad is to model IO actions that are also able to
yield information about their continuing execution on a Chan that can
be polled by the thread responsible for actually displaying test
progress to the user.

My current implementation of this feature is the source of lots of all
of multiparameter type classes and fundeps in the code, but there is a
way to do it in Haskell 98. Thus the only cost of the feature would be
(localised) complexity in the code of what you call the "test
frameworks" that translate e.g. QC properties into the Test data type,
and the "test agents" that actually run the tests.

Without the incremental generation feature, we would just need to make
this sort of change to the interface:

{{{
+data Tests = ATest String Test
+           | Tests String [Tests]

-runTests :: [TestSuite.Test] -> IO ()
+runTests :: [TestSuite.Tests] -> IO ()

class TestOptions t where
+  -- | Return a general description of the sort of test being run,
e.g. "Property" or "Test Case"
+  testTypeName :: t -> String

-test :: QC.Testable prop => String -> prop -> Cabal.Test
+test :: QC.Testable prop => String -> prop -> Cabal.Tests
}}}

Should we add the (more invasive) incremental result generation stuff as well?

Cheers,
Max



More information about the cabal-devel mailing list