[Haskell-cafe] GSoC: Improving Cabal's Test Support

[Thomas Tuegel]

On Thu, Apr 1, 2010 at 9:13 PM, Rogan Creswick <creswick at gmail.com> wrote:
> On Thu, Apr 1, 2010 at 3:52 PM, Thomas Tuegel <ttuegel at gmail.com> wrote:
> There are a few frameworks that provide limited degrees of this
> functionality.  I've recently added to test-framework so that the
> results can be gathered into an xml format that complies with at least
> some (maybe all?) junit xml parsers.
>
> I specifically targeted junit xml so it would be easy to use existing
> continuous integration systems as-is, but the format is not for
> haskell tests.  It would be nice, for example, to see how many
> successful quickcheck inputs were run; and the concept of packages and
> classes had to be munged to work with Haskell modules and test
> groupings.
>
> I need to clean up the code and get it over to Max for review before
> it'll be widely available, but that's just a matter of finding the
> time (possibly next week).

Thanks for pointing this out!  I also noticed that the testrunner
package [1] provides almost the same functionality I described here.
(In fact, it's almost exactly the same on the type level.)  I wouldn't
want to reinvent the wheel, and leveraging one of these existing
solutions would allow me to get a lot more done over the course of the
summer.

I have been somewhat hesitant about using an xml format.  If we want
Cabal to be able to process the test results itself, which is not
strictly necessary but potentially useful, using xml would have a
substantial impact on Cabal's build dependencies.

>> The test suite would be included in the package description file with
>> a stanza such as:
>>
>>> Test
>>>         main-is: Test.hs
>>>         build-depends: foo, QuickCheck, Cabal
>
> I've been thinking about this as well, and I like this general idea,
> but I'm not (yet) convinced it's the best.  That's probably just
> because I'm cautious though :)

Well, let me try to convince you! :)

My intention is to have Cabal handle the 'Test' stanza exactly as if
it were an 'Executable test' stanza.  I think this gives package
authors the most flexibility and greatest ease of use.  It's easy to
use because, if you can write an 'Executable' stanza, you can write a
'Test' stanza.  At the same time, you get all of Cabal's flexibility
in handling executables.  Furthermore, since Cabal will build this
executable as if it were any other, you can write any test suite you
want!  Don't buy in to the idea of a common output format for test
results?  Want to use a different framework?  No problem!  To Cabal,
this is just another executable that it builds, and invokes when you
ask for tests (and one that it knows not to install).

I think ease of use and flexibility are the most important aspects of
this plan.  Part of the goal here is to entice more package authors to
write test suites, which will only happen if the barrier to entry is
low.  At the same time, it would be nice to convert some authors with
existing high-quality test suites, which requires flexibility to work
with their existing tests.

>> At this point, the package author need only run:
>>
>> $ ./Setup configure
>> $ ./Setup build
>> $ ./Setup test
>
> My general feeling has been that Setup is being discouraged in favor
> of using 'cabal <foo>', but I don't have any solid evidence for that
> (and I could very well be wrong!).  They do do slightly different
> things, so I think it's wise to figure out which idiom is most likely
> to be used and work with that.

The Cabal documentation wasn't entirely clear to me, regarding which
is preferred.  In any case, the changes I'm talking about making to
Cabal would happen in 'Distribution.Simple,' and cabal-install seems
to use that, so the improvements should be available to authors and
users regardless of which they use.

-- 
Thomas Tuegel

[1]  http://hackage.haskell.org/package/testrunner