Cabal's detailed test interface

[Thomas Tuegel]

On Wed, Jul 27, 2011 at 7:32 PM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:

> Q: what should 'concurrently' mean for a test group that contains other
> test groups? Suppose a test group asks for serial execution, does that
> mean it's ok to run tests in a sub-group in parallel with each other, so
> long as -- as a group -- they get run serially with respect to the top
> group?

Yes, what you describe was my intent. I should document this more
carefully in the module.

>> * The 'Finished' constructor of 'Progress' now contains the options
>> used to run the test in addition to the test result. Without returning
>> the options, it would be difficult to extract the RNG seed used to run
>> a test.
>
> I'm not sure if I believe this one. Any test agent that cares about
> reproducing RNG seeds can just pick the RNG seed itself. It can tell
> what options are RNGs by looking for options with option type
> OptionRngSeed.

You're probably right. I will remove the options from Finished for now
while we validate the design. If it turns out to be necessary, we
could easily re-add it before committing the final version of the
interface.

>> * A detailed test suite module now exports the symbol 'test :: IO
>> Tests'.
>
> Perhaps it should be tests, plural :-)

It is, that's a typo in the patch log >_<

>> This enables the use of IO to enumerate the tests in a group,
>> suggested by Duncan as a way to accommodate the GHC test suite.
>
> Speaking of IO, do we ever need to deal with test suites that need
> imperative setup and teardown for sets of tests? Individual tests can do
> setup and teardown via the Progress which embeds IO. Does HUnit allow IO
> for sets/groups of tests?

I can see the utility of this. It would be simple enough to have Group
take an (IO [Tests]) instead of just [Tests].

>> > data OptionType
>> >     = OptionFile
>> >         { optionFileMustExist   :: Bool
>> >         , optionFileIsDir       :: Bool
>> >         , optionFileExtensions  :: [String]
>> >         }
>> >     | OptionString
>> >         { optionStringMultiline :: Bool
>> >         , optionStringDescription :: String
>
> What is this description? We've already got an option description.

You're right, of course; it serves no purpose. I think I added it in a
moment of confusion.

>> As always, your comments are appreciated. Thanks!
>
> Ok, so if we're into the phase of validating our design, apart from
> seeing if we could adapt test-framework to fit this interface, we should
> also check on the output side that we have enough info / detail to be
> able to generate JUnit XML or whatever other is the format de jour for
> describing results of test suites.

I'm looking at the JUnit schema right now. From the Dist.TestSuite
side, it looks like we have all the information needed. However, it
isn't possible to directly convert the logs made by "cabal test" to
JUnit XML, as I previously claimed it was.

The incompatibility is mainly due to a lack of granularity in
capturing the standard output and error which was acceptable with the
executable-stdio suites, but may not be acceptable here. This will
require me to think about overhauling the default test runner in
Dist.Simple.Test.

-- 
Thomas Tuegel