Cabal's detailed test interface

[Thomas Tuegel]

Greetings once again!

I have attached an updated patch with the requested changes to the
test interface. (I apologize for the delay; it was caused by the
unfortunate conspiracy of moving, not having internet access, and
qualifying exams. All of these being done, I will be more responsive
on Cabal-related things.)

Here is a summary of the patches, with some commentary:

This is the patch from before:
Tue Jul 26 17:29:23 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * New detailed test suite interface.
  This patch implements a new interface for detailed test suites based
on Duncan's
  proposal. This implementation differs from his in a few ways:
  * The constructors of 'Tests' have been renamed. I think 'TestGroup' and
    'ExtraTestOptions' are redundant: it is clear from context what
sort of Group
    and what sort of ExtraOptions are being considered and qualified imports can
    resolve any name conflicts. Group and ExtraOptions have the advantage of
    improving the brevity of pattern matches on Tests, which are used very often
    in D.S.Test.
  * The 'concurrentSafe :: Bool' field of TestInstance has become the
    'concurrently' field of Group, allowing package and test framework authors
    greater control over concurrency.
  * The 'Finished' constructor of 'Progress' now contains the options
used to run
    the test in addition to the test result. Without returning the options, it
    would be difficult to extract the RNG seed used to run a test.
  * A detailed test suite module now exports the symbol 'test :: IO Tests'. This
    enables the use of IO to enumerate the tests in a group, suggested by Duncan
    as a way to accomodate the GHC test suite.

The name change Johan suggested:
Fri Jul 29 10:02:38 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Renamed Distribution.TestSuite.Tests to Test.

Forgot to remove ExistentialQuantification.
Fri Jul 29 10:10:01 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Removed obsolete LANGUAGE pragma from Distribution.TestSuite.

Fri Jul 29 10:42:49 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Cleaned for warnings in D.TestSuite and D.S.Test.

Duncan pointed out that this field is redundant:
Fri Jul 29 10:48:12 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Removed 'optionStringDescription' from OptionType.

Johan suggested having the symbol exported by test modules be "tests
:: IO [Test]" rather than "tests :: IO Test". Formerly, all the tests
in a suite would have to be wrapped in a named Group, and as Johan
pointed out, naming things is a hard problem.
Thu Sep  1 13:26:06 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Changed detailed test exported type to [Test].

After the last patch, the test cases in a suite aren't structured as a
simple tree, so the 'OptionTree' type I devised for reading options
from a log file and reapplying them doesn't make much sense. I opted
to remove it for the reasons explained below.
Tue Sep  6 16:16:11 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Clean D.S.Test for unused symbols.
  The unused declarations were all related to replaying test suites with logged
  options, but the command-line option for this feature has been
disabled for some
  time. Changing the detailed test suite type to expect "tests :: IO [Test]"
  instead of "tests :: IO Test" made the old method of replaying options clumsy.
  Since it was already disabled, I chose to remove it, rather than rewrite it
  again.

Per Duncan's suggestion:
Tue Sep  6 16:55:57 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Removed Options from Finished.

The documentation makes explicit the fact that the 'concurrently'
property is not inherited.
Tue Sep  6 16:56:00 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Improved documentation for 'concurrently' field of Test.

The convenience function Johan requested:
Tue Sep  6 18:36:32 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Added 'testGroup' to D.S.TestSuite.

Since most of the major changes are done, I updated the user manual
(so it won't fall through the cracks later).
Tue Sep  6 18:49:31 CDT 2011  Thomas Tuegel <ttuegel at gmail.com>
  * Updated user manual for new detailed interface.


There's still the issue of a trial implementation of the interface:

On Wed, Jul 27, 2011 at 7:32 PM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:
> Perhaps one of us should have a go at seeing if we can make the ghc test
> suite fit this interface. Perhaps a project for the hackathon...

I had a look at the GHC test suite. It looks to me like the Python
test runner would have to be replaced with some Haskell code to make
it fit the detailed interface. I don't have any experience with GHC or
its test suite, though; there may be an easier way that I'm not
seeing.

-- 
Thomas Tuegel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cabal-detailed-tests-2.dpatch
Type: application/octet-stream
Size: 58197 bytes
Desc: not available
URL: <http://www.haskell.org/pipermail/cabal-devel/attachments/20110907/fab5efc6/attachment-0001.obj>