[Haskell-cafe] What is the state of the art in testing code generation?

[Tom Ellis]

The first impediment to using QuickCheck is that my EDSL is typed.  I don't
know how to generate random terms in the presence of types.

The second impediment is that, although I can indeed test trivial
properties, I don't know how to test functionality to its full level of
sophistication.  How can I test a left join, for example, without
reimplementing left join functionality within Haskell?  (Indeed I may end up
doing this, but would prefer to find an easier path ...)

Tom

On Mon, Jul 14, 2014 at 11:26:14AM +0200, Vo Minh Thu wrote:
> I think there is still some opportunity for something like QuickCheck
> in your case. Certainly you can use QC to generate
> expressions/statements in your EDSL. Then if you can also generate
> schemas and data you should be able to write down some properties, for
> instance that some class of queries on empty tables should return no
> rows.
> 
> A special case of this, and a more specific example, is very similar
> to the introductory examples to QC: it must be possible to retrieve a
> row after inserting it in an empty table, or deleting it after
> inserting it must leave the table unchanged.
> 
> Even if you don't go as far as writing properties that involves the
> schema/data generation, generating arbitrary valid AST that must
> compile successfully to SQL is interesting.

> 2014-07-11 19:58 GMT+02:00 Tom Ellis
> <tom-lists-haskell-cafe-2013 at jaguarpaw.co.uk>:
> > I am implementing an EDSL that compiles to SQL and I am wondering what is
> > the state of the art in testing code generation.
> >
> > All the Haskell libraries I could find that deal with SQL generation are
> > tested by implementing multiple one-off adhoc queries and checking that when
> > either compiled to SQL or run against a database they give the expected,
> > prespecified result.
> >
> >  * https://github.com/prowdsponsor/esqueleto/blob/master/test/Test.hs
> >  * https://github.com/m4dc4p/haskelldb/blob/master/test/TestCases.hs
> >  * https://github.com/yesodweb/persistent/blob/master/persistent-test/SumTypeTest.hs
> >
> > I couldn't find any tests for groundhog.
> >
> >  * https://github.com/lykahb/groundhog
> >
> > I also had a look at Javascript generators.  They take a similar adhoc,
> > one-off approach.
> >
> >  * https://github.com/valderman/haste-compiler/tree/master/Tests
> >  * https://github.com/faylang/fay/tree/master/tests
> >
> > Is this the best we can do in Haskell?  Certainly it seems hard to use a
> > QuickCheck/SmallCheck approach for this purpose.  Is there any way this kind
> > of testing can be automated or made more robust?