Cabal's detailed test interface

[Duncan Coutts]

On Tue, 2011-07-26 at 19:38 -0500, Thomas Tuegel wrote:
> Greetings interested Cabal users and developers!
> 
> I am pleased to present another revision of Cabal's detailed test
> interface for your scrutiny.

First of all, thanks for going over this again. I think we're getting
closer.

Probably the next step, as Johan alludes to, is trying to implement this
interface (e.g. a prototype adaptor for test-framework).

> I have attached a patch for the Cabal repository to this message so
> you can try the new interface and test runner for yourselves, but the
> interface from Distribution.TestSuite is included below with a few of
> my comments.
> 
> This interface is based very heavily on Duncan's proposal from our
> last round of discussion. I indicated deviations by comments in the
> source, but the commit log is a good summary of the changes:
> 
> * The constructors of 'Tests' have been renamed. I think 'TestGroup'
> and 'ExtraTestOptions' are redundant: it is clear from context what
> sort of Group and what sort of ExtraOptions are being considered and
> qualified imports can resolve any name conflicts. Group and
> ExtraOptions have the advantage of improving the brevity of pattern
> matches on Tests, which are used very often in D.S.Test.

Having 'ExtraTestOptions' as another Tests constructor is a good idea.

> * The 'concurrentSafe :: Bool' field of TestInstance has become the
> 'concurrently' field of Group, allowing package and test framework
> authors greater control over concurrency.

Yes, fine. That indicates better the scope of what it is that can be run
concurrently. And in case it wasn't clear before, you and Max are right
about how the distinction ought to be concurrency rather than purity.

Q: what should 'concurrently' mean for a test group that contains other
test groups? Suppose a test group asks for serial execution, does that
mean it's ok to run tests in a sub-group in parallel with each other, so
long as -- as a group -- they get run serially with respect to the top
group?

> * The 'Finished' constructor of 'Progress' now contains the options
> used to run the test in addition to the test result. Without returning
> the options, it would be difficult to extract the RNG seed used to run
> a test.

I'm not sure if I believe this one. Any test agent that cares about
reproducing RNG seeds can just pick the RNG seed itself. It can tell
what options are RNGs by looking for options with option type
OptionRngSeed.

Am I missing anything?

On a related topic, I was looking back at an older conversation between
you and Max. Max was criticising our GetOpt-style option parsing design
of the time. He said:
        
        We are unlikely to do a good job at covering all use cases here.
        We could:
             1. Accept that, and just stick with only providing
                --foo=bar arguments
             2. Support more forms of arguments by using the GetOpt
                Option data type or our own implementation of it
             3. Just delegate everything to the test provider by having
                them provide a (run :: p -> [String] -> IO (Either
                String Result)), where errors in the arguments are
                reported with Left "Error information". Optionally we
                could add a (usage :: IO String) function for retrieving
                usage information for the command line.

I think I'm satisfied that our current design addresses these
criticisms. We're now going more or less with option 3, in that the test
provider is ultimately responsible for the parsing, and it just uses
strings. But we're then augmenting that with a little bit of declarative
presentation information so that test runners can provide reasonable
user interfaces / forms for data input, while still leaving the data
parsing / validation up to the test provider.

> * A detailed test suite module now exports the symbol 'test :: IO
> Tests'.

Perhaps it should be tests, plural :-)

> This enables the use of IO to enumerate the tests in a group,
> suggested by Duncan as a way to accommodate the GHC test suite.


Speaking of IO, do we ever need to deal with test suites that need
imperative setup and teardown for sets of tests? Individual tests can do
setup and teardown via the Progress which embeds IO. Does HUnit allow IO
for sets/groups of tests?

I'm just thinking of a recent little project we had at work where we
were testing DB connections and had to populate a database with some
sample data before running any of a group of tests. (Admittedly this was
a performance rather than correctness test, but it almost certainly
transfers to correctness tests.)

> > module Distribution.TestSuite
> >     ( TestInstance(..)
> >     , OptionDescr(..)
> >     , OptionType(..)
> >     , Tests(..)
> >     , Options
> >     , Progress(..)
> >     , Result(..)
> >     ) where
> >
> > data TestInstance = TestInstance
> >     { run       :: IO Progress      -- ^ Perform the test.
> >     , name      :: String           -- ^ A name for the test, unique within a
> >                                     -- test suite.
> >     , tags      :: [String]         -- ^ Users can select groups of tests by
> >                                     -- their tags.
> >     , options   :: [OptionDescr]    -- ^ Descriptions of the options recognized
> >                                     -- by this test.
> >     , setOption :: String -> String -> Either String TestInstance
> >         -- ^ Try to set the named option to the given value. Returns an error
> >         -- message if the option is not supported or the value could not be
> >         -- correctly parsed; otherwise, a 'TestInstance' with the option set to
> >         -- the given value is returned.
> >     }
> >
> > data OptionDescr = OptionDescr
> >     { optionName        :: String
> >     , optionDescription :: String       -- ^ A human-readable description of the
> >                                         -- option to guide the user setting it.
> >     , optionType        :: OptionType
> >     , optionDefault     :: Maybe String
> >     }
> >   deriving (Eq, Read, Show)
> >
> > data OptionType
> >     = OptionFile
> >         { optionFileMustExist   :: Bool
> >         , optionFileIsDir       :: Bool
> >         , optionFileExtensions  :: [String]
> >         }
> >     | OptionString
> >         { optionStringMultiline :: Bool
> >         , optionStringDescription :: String

What is this description? We've already got an option description.

If it's an opaque string (as far as test runners are concerned) then it
doesn't help them with presentation. I'd expect a hypothetical
interactive GUI for a test suite to just validate string fields on the
fly using setOption.

> As always, your comments are appreciated. Thanks!

Ok, so if we're into the phase of validating our design, apart from
seeing if we could adapt test-framework to fit this interface, we should
also check on the output side that we have enough info / detail to be
able to generate JUnit XML or whatever other is the format de jour for
describing results of test suites.

Perhaps one of us should have a go at seeing if we can make the ghc test
suite fit this interface. Perhaps a project for the hackathon...

Duncan