Any remaining test patches?

[Duncan Coutts]

On 21 May 2011 15:20, Duncan Coutts <duncan.coutts at googlemail.com> wrote:

> So we either have a individual test or a group of tests (which
> themselves can contain groups etc). I'm sure this is nice for
> displaying in a UI what tests can be run and displaying results of a
> run. Imagine an interactive UI that lets the user select a group of
> tests to be run, or an html report showing results by group.
>
> We could do something like this:
>
> data Tests = Test TestInstance
>           | TestGroup TestName [Tests]
>

Actually, we need more than this. We need to be able to do IO to
enumerate the tests in the group.

Consider the ghc tests suite. It is an interesting and reasonably
large scale example. I think we ought to make sure that our test suite
interface enables us to wrap the current ghc testsuite without having
to do a major rewrite.

The ghc test suite is implemented as a whole bunch of files arranged
into directories with a few test-specific scripts in a few places. So
it matches the heirarchy notion well enough but to list the tests in
each group (in each dir in the ghc testuite case) means doing IO to
list the files in the directory and pull out the ones that are
recognised as test cases.

The ghc example also makes me thing that we want a bit more
declarative info that could be presented in a web/gui interface. I
think we want optional descriptions on each group of tests, or on
individual tests. Many of ghc tests have a notion of "way", that is
you can run the same test with normal, ghci, profiled, dynamic etc
ways.

Is there anything we can do to expose that information? I don't think
we want to make individual test instances paramaterised, but perhaps
we can do something slightly more detailed than TestGroup to indicate
that some test instances are related to each other in certain ways.
For the ghc "way" example, suppose we annotated each test with its
way. A UI could then allow things like filtering on these attributes.
For ghc's testsuite the use case is to let users do things like only
run tests the ghci way, or exclude the opt way. In the ghc test suite
the current driver does things like grouping failing tests so that the
test name is listed only once, and each failing way is listed
together, e.g.:

  T5636 (ghci, opt, prof)

So what if in addition to this system of test options (inputs) we had
a similar declarative system for describing test attributes. What
might it look like and how could test agents expose this so that users
can make use of it?

Duncan