Any remaining test patches?

[Thomas Tuegel]

On Sat, May 21, 2011 at 9:20 AM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:
> data TestInstance
>   = TestInstance {
>       run            :: IO TestResult
>       name           :: String,
>
>       concurrentSafe :: Bool,
>       expectedFail   :: Bool,
>
>       options        :: [OptionDescr]
>       setOption      :: String -> String -> Either String TestInstance
>     }

I've been working on this new interface and a test runner for it, and
I've made a discovery about this property of "exclusivity"
(alternately, "concurrent safety") that I think I should share.

Exclusivity is not a property of individual tests, but of test groups.
If you think carefully about the (admittedly pathological) case of a
test suite with only one test case, you see that it doesn't matter
whether it is marked as exclusive or not: it will be run both
concurrently and sequentially with all the other tests. Exclusivity
only exists when there is more than one test.

I am working on a prototype interface that is essentially a replica of
what Duncan outlined here, but I have moved the "concurrentSafe" field
to the TestGroup constructor. Not only is this "more correct," but it
buys a lot of expressiveness. With exclusivity as a property of
individual tests, test runners must simply run all the unsafe tests in
sequence. In reality, though, it is likely that there will be groups
of tests which can be run concurrently. For example, suppose we have
four tests which conflict with each other in pairs, i.e., tests 1 and
2 cannot be run at the same time and neither can tests 3 and 4: we can
still run tests 1 and 3 in parallel and tests 2 and 4 in parallel.
With "concurrentSafe" as a field of TestGroup, this dependency can be
expressed and test runners can maximize concurrency.

-- 
Thomas Tuegel