Automated tests for the libraries?

[Neil Mitchell]

Hi Jeffrey,

Sorry for the delay in replying - I've been busy with other stuff.

> I've been working on getting test running Cabalized and finally have a
> patch to send. Extracting tests from the source code is, I think, an
> independent problem.

The code for FilePath is going to be including with GHC 6.6.1 which
comes out very shortly, so I'd like to put off adding things to Cabal
to run the tests until after that. Every change at this time is a
little dangerous so I want to keep them minimal.

However, after GHC 6.6.1 comes out, and we have a bit more breathing
space, I'd love to have the Setup.hs tests run the tests, rather than
having a separate .bat file do it. Ideally I want to be consistent
with the other packages, so maybe Ian can advise as to how testing
should look like? What does testing need to do to get into the GHC
buildbot test scripts?


> http://jeffrey.yasskin.info/darcs/filepath/SingleFileOps.hs has some
> operations that I'd like to add to Cabal itself. With some more
> experience writing tests like this, we'll definitely want to extract
> most of this Setup.hs into a shared module.

You'll want to look at the Cabal lists for what of the SingleFileOps
can go into there.


> The basic idea is that I want to base the tests as much as possible on
> production code, but I may need to #define TESTING or some other
> symbol in a particular file to get that file's tests to run. So each
> file "foo.hs" may be compiled with the 'tag' "foo" (practically, into
> the dist/build/foo directory) with -DTESTING, and then its test is run
> in the context of ["foo", ""] (look for "foo" object files, and then
> the results of `Setup.hs build`). Putting testing builds in a
> different place also makes sure that they don't get installed or
> released accidentally.

All sounds sensible.

> What would be the best way to proceed? Is this system overkill and I
> should start over with something simpler (either a single -DTESTING
> build tree or just `runhaskell -DTESTING`)? Should I clean up
> SingleFileOps and submit it to Cabal? Should I take what I have and
> apply it to base?

You'd have to ask the Cabal people how much test stuff they want in
Cabal, and perhaps have a separate tool that does test extraction if
they'd rather it was kept minimal.

Thanks

Neil

>
> On 3/18/07, Neil Mitchell <ndmitchell at gmail.com> wrote:
> > Hi
> >
> > > > This gives a list of properties, which are automatically extracted
> > > > from the documentation, and checked using QuickCheck. All the tool is
> > > > in the repo. If anyone wanted to generalise the code in there, it
> > > > would be handy!
> > >
> > > Neil, your extractor looks cool, but I wonder how it'll work with
> > > modules like Data.Set whose (commented out) tests define some
> > > Arbitrary instances and some helper functions in addition to the
> > > actual properties. My first inclination would have been to define the
> > > properties in code, perhaps "#ifdef TESTING"ed out, and try to teach
> > > Haddock to pull them into the documentation. But yours is definitely
> > > simpler and probably worth pursuing until it obviously stops working.
> >
> > FilePath also has that issue, I do:
> >
> > module FilePath(
> > #ifdef TESTING
> >     properties and instances for testing
> > #endif
> >
> >    ) where
> >
> > Then I just define TESTING when running the test. This has the
> > advantage that the tests at least always compile, even as I make other
> > changes. In reality, FilePath only requires a couple of things (the
> > drive properties) to be exported for testing but not for real use.
> >
> > I don't see much complication in extending my approach. In fact, I
> > suspect that my FilePath test extractor is overly complex, since it
> > allows:
> >
> > Windows: property
> > Posix: property
> > property
> >
> > For Windows properties it added System.FilePath.Windows.functions,
> > ditto for Posix, and for normal ones it generates the test twice, once
> > with Windows and once with Posix. It also detects if a property has
> > any free variables, and if not changes it to a single assertion,
> > rather than a quickcheck property.
> >
> > I think it is very handy, and integrating it with Cabal and removing
> > the special cases that FilePath requires is probably well worth the
> > effort! I would certainly like such a tool...
> >
> > Thanks
> >
> > Neil
> >
>
>
> --
> Namasté,
> Jeffrey Yasskin
>
>