Next steps for "cabal test" feature

[Thomas Tuegel]

Hello!

Max has some excellent suggestions for detailed test interface:

The first subject we've been discussing is the orthogonality of the
pure/impure distinction from safe/unsafe.  Specifically, the current
interface makes the pure/impure distinction, but we really care about
the safe/unsafe distinction.  There are, e.g., safe, impure tests.
One thing to do would be to change the interface so there is only one
"Testable" class, allow tests in it to run in IO, and provide some
kind of combinator for denoting safe/unsafe tests.

On Mon, Nov 15, 2010 at 6:47 AM, Max Bolingbroke
<batterseapower at hotmail.com> wrote:
> 2) Even better, we could not support unsafe tests at all. They should
> be extremely uncommon, and if the user really wants to do that they
> can implement it themselves by wrapping all tests in a multiple-reader
> single-writer lock: have safe tests aquire a reader lock and unsafe
> ones the writer lock during execution. This strategy can even be
> implemented by combinator of a type such as :: [(Safe, Test)] -> IO
> [Test] (where type Safe = Bool) outside of the cabal test library, so
> that the user wouldn't need to write the locks themselves.

One other thing we've been discussing is what to do about TestOptions.
The fallout of this discussion is essentially that everything
TestOptions is trying to do, GetOpt does better.

On Mon, Nov 15, 2010 at 6:47 AM, Max Bolingbroke
<batterseapower at hotmail.com> wrote:
> If this is our goal, we should probably choose a different API, closer
> to that provided by GetOpt or similar. The proposed API only supports
> arguments of the form --foo=bar, where bar is a string. There are many
> other common patterns:
> * --baz (with no argument)
> * -v vs. --verbose (short form arguments)
> * --meh={a,b,c} (arguments whose values are drawn from an enumeration
> -- you could imagine e.g. populating a combo box with these)
>
> We are unlikely to do a good job at covering all use cases here. We could:
> 1. Accept that, and just stick with only providing --foo=bar arguments
> 2. Support more forms of arguments by using the GetOpt Option data
> type or our own implementation of it
> 3. Just delegate everything to the test provider by having them
> provide a (run :: p -> [String] -> IO (Either String Result)), where
> errors in the arguments are reported with Left "Error information".
> Optionally we could add a (usage :: IO String) function for retrieving
> usage information for the command line.

Option 1 is what we're unsatisfactorily doing now. Option 2 would be
to have a method such as "options :: [OptDescr (t -> t)]" in class
Testable. Option 3 is to delegate option parsing to the test provider.
I prefer option 2: compared to option 3, it enforces standardization
more rigidly and takes some of the work off test provider authors.

Thoughts? I thought if we could come to a consensus here, I can have
some patches by the end of the week.

Thanks!

-- 
Thomas Tuegel