Next steps for "cabal test" feature

[Thomas Tuegel]

First, let me apologize for taking so long to respond; school is
keeping me quite busy.

On Mon, Oct 11, 2010 at 5:13 PM, Max Bolingbroke
<batterseapower at hotmail.com> wrote:
> OK, I've done a bit of a review.
>
> Features of test-framework not in "cabal test":
>  * Test groups
>  * Ability to see summary of test results by test type (e.g. 2 failed
> properties, 5 failed test cases)

I agree that these features should be in cabal test.

>  * Incremental generation of test results (so you can e.g. see
> interactively the # of properties run so far while a property is still
> executing)

I think this is a good idea.  It is surely worth the complication, but
I don't think of it as a top priority, at least until some other
issues are resolved.

> Features of "cabal test" not in test-framework:
>  * The result data type lets you distinguish between "errors" and "failures"
>  * More extensible options type (i.e. it *is* actually extensible :-)
>
> Questionable features of "cabal test":
>  * I'm not too sure what the "options" member of "TestOptions" gives
> you - after all, the test has
>   to do it's own parsing of the [(String, String)] Options list, in
> "check" and "run"/"runM" - so Cabal
>   doesn't really get to use this information

I want to rework TestOptions anyway, based on my own distaste for it
and the several criticisms it received at the Implementors Workshop.
(More on this below.)

>  * Separation of pure/impure tests - what do you want to use this for?
> Good unit tests should be isolated from each other (no dependencies).
> Given this constraint, we can safely run IO-type unit tests in
> parallel with each other and so on - so it seems that knowing they are
> pure is not of great benefit?

Good unit tests _should_ be isolated, but I'm not sure we can always
count on good ones. Truly pure tests can't help but be isolated,
though. You are right that the majority of the time, it won't matter,
but I think the overhead of keeping the distinction around is low
enough to be worth it.

I also want to discuss what we're going to do with the TestOptions
type.  The criticisms it received at HIW all centered around the
"type-class as dictionary" pattern that so many Haskellers find
distasteful.  In my own opinion, the interface would be awkward to use
and didn't provide adequate protections against setting invalid
options.  I want to propose the following interface instead of what
currently exists:

> -- | Abstraction of a field for an internal type.
> data OptionField i = OptionField {
>                                  -- | The name of the field. Used to construct the 'TestOptions'.
>                                    name :: String
>                                  -- | The type of the field. Used to construct the 'TestOptions'.
>                                  , fieldType :: TypeRep
>                                  -- | Read and set this field in the internal type, returning the new data if the input was valid.
>                                  , setter :: i -> String -> Maybe i
>                                  -- | Show this field from the internal type.
>                                  , getter :: i -> String
>                                  }
>
> data TestOptions p = TestOptions { -- | Set the field named in the first argument to the value
>                                    -- read from the second. Return 'Nothing' if the field does
>                                    -- not exist or the value cannot be read.
>                                    set :: String -> String -> Maybe (TestOptions p)
>                                    -- | Return the value of the named field, if it exists.
>                                 , get :: String -> Maybe String
>                                 , fields :: [(String, TypeRep)]
>                                  }
>
> mkTestOptions :: [OptionField i] -> i -> TestOptions p
>
> class Testable t where
>     defaultOptions :: IO (TestOptions t)
>     testTypeName :: t -> String
>     testName :: t -> String
>
> class Testable p => PureTestable p where
>     run :: p -> TestOptions p -> Result
>
> class Testable i => ImpureTestable i where
>     runM :: i -> TestOptions i -> Result

If we agree that this would be an improvement, I will submit a patch.

-- 
Thomas Tuegel