Any remaining test patches?

[Thomas Tuegel]

On Wed, Jun 29, 2011 at 2:38 AM, Johan Tibell <johan.tibell at gmail.com> wrote:
> On Tue, Jun 28, 2011 at 7:28 PM, Thomas Tuegel <ttuegel at gmail.com> wrote:
>> Exclusivity is not a property of individual tests, but of test groups.
>> If you think carefully about the (admittedly pathological) case of a
>> test suite with only one test case, you see that it doesn't matter
>> whether it is marked as exclusive or not: it will be run both
>> concurrently and sequentially with all the other tests. Exclusivity
>> only exists when there is more than one test.
>
> I don't think this is necessarily the case. If you put concurrentSafe
> on a test you'd then partition tests like so
>
>    (safe, unsafe) = partition isConcurrentSafe tests
>
> and then run all of the safe tests (possibly in parallel) and then all
> the unsafe tests one by one.

Your suggestion is guaranteed to be safe, but someone will look at the
test interface and think the following is acceptable: Partition the
safe and unsafe tests; run the safe tests all in parallel and, at the
same time, run the unsafe tests one by one. This hypothetical
programmer might rationalize this reading by saying "If it isn't safe
to run concurrentSafe tests concurrently with _any_ other test, even
an unsafe one, then concurrentSafe tests aren't very safe, are they?"

This isn't our interpretation, so there will have to be an officially
sanctioned parallelization scheme in the documentation. I don't
dispute that a simple bit of documentation fixes the ambiguity (as
long as future test runner programmers read the docs very carefully),
but I would prefer to modify the data structure. If groups are marked
concurrentSafe instead of tests, there is absolutely no ambiguity
about which tests can safely be run concurrently.

As an example, the parallelization scheme you came up with would look like this:

TestGroup
    { groupName = "AllTests"
    , concurrentSafe = False
    , groupTests =
        [ TestGroup
            { groupName = "SafeTests"
            , concurrentSafe = True
            , groupTests = [...]
            }
        , TestGroup
            { groupName = "UnsafeTests"
            , concurrentSafe = False
            , groupTests = [...]
            }
        ]
    }

whereas my alternative interpretation would be:

TestGroup
    { groupName = "AllTests"
    , concurrentSafe = True
    , groupTests =
        [ ... ]
    }

(Only the third line changes.) With concurrentSafe at the Group level,
the difference is clear. More importantly, this gives package and test
framework authors more power to express how their tests can be run
without increasing the complexity of test runners.

-- 
Thomas Tuegel