Any remaining test patches?

[Johan Tibell]

On Sat, May 21, 2011 at 7:36 PM, Duncan Coutts
<duncan.coutts at googlemail.com> wrote:
> Actually, we need more than this. We need to be able to do IO to
> enumerate the tests in the group.
>
> Consider the ghc tests suite. It is an interesting and reasonably
> large scale example. I think we ought to make sure that our test suite
> interface enables us to wrap the current ghc testsuite without having
> to do a major rewrite.
>
> The ghc test suite is implemented as a whole bunch of files arranged
> into directories with a few test-specific scripts in a few places. So
> it matches the heirarchy notion well enough but to list the tests in
> each group (in each dir in the ghc testuite case) means doing IO to
> list the files in the directory and pull out the ones that are
> recognised as test cases.
>
> The ghc example also makes me thing that we want a bit more
> declarative info that could be presented in a web/gui interface. I
> think we want optional descriptions on each group of tests, or on
> individual tests. Many of ghc tests have a notion of "way", that is
> you can run the same test with normal, ghci, profiled, dynamic etc
> ways.
>
> Is there anything we can do to expose that information? I don't think
> we want to make individual test instances paramaterised, but perhaps
> we can do something slightly more detailed than TestGroup to indicate
> that some test instances are related to each other in certain ways.
> For the ghc "way" example, suppose we annotated each test with its
> way. A UI could then allow things like filtering on these attributes.
> For ghc's testsuite the use case is to let users do things like only
> run tests the ghci way, or exclude the opt way. In the ghc test suite
> the current driver does things like grouping failing tests so that the
> test name is listed only once, and each failing way is listed
> together, e.g.:
>
>  T5636 (ghci, opt, prof)
>
> So what if in addition to this system of test options (inputs) we had
> a similar declarative system for describing test attributes. What
> might it look like and how could test agents expose this so that users
> can make use of it?

I'm worried about generalizing from a single instance (i.e. GHC).

If we'd like to add some flexibility we could allow tests to carry
arbitrary tags:

class TestInstance a where
    tags :: a -> [String]

Test agents could use tags to group tests in different ways. We could
add  support for tags in Cabal's test runner so that they can be used
to select which tests to run. For example,

cabal test --test-tags="smoke -flaky"

could run all non-flaky smoke tests.

Johan