Next steps for "cabal test" feature

[Max Bolingbroke]

On 12 November 2010 01:16, Thomas Tuegel <ttuegel at gmail.com> wrote:
> First, let me apologize for taking so long to respond; school is
> keeping me quite busy.

No problem.

>>  * Separation of pure/impure tests - what do you want to use this for?
>> Good unit tests should be isolated from each other (no dependencies).
>> Given this constraint, we can safely run IO-type unit tests in
>> parallel with each other and so on - so it seems that knowing they are
>> pure is not of great benefit?
>
> Good unit tests _should_ be isolated, but I'm not sure we can always
> count on good ones. Truly pure tests can't help but be isolated,
> though. You are right that the majority of the time, it won't matter,
> but I think the overhead of keeping the distinction around is low
> enough to be worth it.

I still don't think it's worth it. There may be a valuable distinction
to make between "tests safe for running concurrently with other tests"
and "tests unsafe for running concurrently with other tests". It is
true that:

"Test is pure" ==> "Test safe for running concurrently"

But the reverse direction of the implication is not true: some impure
tests are safe for running concurrently. So impure/pure is the wrong
distinction: it lumps safe impure tests in with unsafe impure tests. I
would prefer that:
1) If we care about supporting unsafe tests, we provide a flag or
SafeTestable/UnsafeTestable type class split
2) Even better, we could not support unsafe tests at all. They should
be extremely uncommon, and if the user really wants to do that they
can implement it themselves by wrapping all tests in a multiple-reader
single-writer lock: have safe tests aquire a reader lock and unsafe
ones the writer lock during execution. This strategy can even be
implemented by combinator of a type such as :: [(Safe, Test)] -> IO
[Test] (where type Safe = Bool) outside of the cabal test library, so
that the user wouldn't need to write the locks themselves.

> I also want to discuss what we're going to do with the TestOptions
> type.  The criticisms it received at HIW all centered around the
> "type-class as dictionary" pattern that so many Haskellers find
> distasteful.  In my own opinion, the interface would be awkward to use
> and didn't provide adequate protections against setting invalid
> options.  I want to propose the following interface instead of what
> currently exists:

OK, so what are we actually trying to achieve with TestOptions? As I
see it, the reason we are exposing this information at all (as opposed
to just having run/runM consume a [String]) is so that we can do
command line argument parsing on behalf of what I call the test
"provider". We could also use the infrastructure to populate a GUI
with a list of test provider options, and generate usage information.

If this is our goal, we should probably choose a different API, closer
to that provided by GetOpt or similar. The proposed API only supports
arguments of the form --foo=bar, where bar is a string. There are many
other common patterns:
* --baz (with no argument)
* -v vs. --verbose (short form arguments)
* --meh={a,b,c} (arguments whose values are drawn from an enumeration
-- you could imagine e.g. populating a combo box with these)

We are unlikely to do a good job at covering all use cases here. We could:
1. Accept that, and just stick with only providing --foo=bar arguments
2. Support more forms of arguments by using the GetOpt Option data
type or our own implementation of it
3. Just delegate everything to the test provider by having them
provide a (run :: p -> [String] -> IO (Either String Result)), where
errors in the arguments are reported with Left "Error information".
Optionally we could add a (usage :: IO String) function for retrieving
usage information for the command line.

My preferred option is 3. as it is the lightest weight solution, and
doesn't require auxiliary OptionField/TestOptions data types. What do
you think? Do we lose something important by this choice?

Cheers,
Max