Cabal's detailed test interface

[Duncan Coutts]

On Tue, 2011-08-16 at 14:23 -0500, Thomas Tuegel wrote:

> Johan and I are talking about renaming the Tests type to just "Test"
> to be more consistent about pluralization. So, also incorporating the
> changes I think you are suggesting to the Group type, it would look
> like:
> 
> data Test
>     = Test TestInstance
>     | Group
>         { groupName     :: String
>         , concurrently  :: Bool
>         , groupTests    :: [Test]
>         , preGroup      :: IO ()
>         , postGroup     :: IO ()
>         }
>     | ExtraOptions [OptionDescr] Test

I think leave off the pre/post IO for now. It's not going to be really
usable like that. It'd have to be something horrible like:

data Test
    = Test (TestInstance a)
    | forall a. Group
         { groupName     :: String
         , concurrently  :: Bool
         , groupTests    :: [Test a]
         , preGroup      :: IO a
         , postGroup     :: a -> IO ()
         }

data TestInstance a = TestInstance
       { run       :: a -> IO Progress

since the individual tests within a group need a handle to whatever was
done in the pre/setup phase. For example imagine a testsuite for HDBC.
Testing the backends requires creating a connection to a database,
populating it with a bunch of test data, running one or more tests on
that test db, and then cleaning and closing the db at the end of the
tests.

It's tricky because we want to enumerate all the tests without running
setup code, but to run a test we'd have had to do all the setup and pass
any results in to the tests.

So I don't suggest we tackle this in full right now. We have a system
with versioned test suite interfaces for a reason, and that's so we can
change and add stuff later.

Everything else seems fine. Lets try it out.


Duncan