[Haskell-cafe] Storing failing testcases for QuickCheck

[wren ng thornton]

On 4/23/11 8:02 AM, Sönke Hahn wrote:
> Hi!
>
> I would like to have a library that would allow to use QuickCheck in the
> normal manner, but it would save test data for failing properties on the
> filesystem (maybe using the shiny new acid-state?). On consecutive test
> runs, the saved test data would be used first (before generating new
> arbitrary test data). I think, this would come in quite handy when doing
> test driven development.
>
> Any comments on this? Does something like this already exist?

The closest I can think of is using HUnit on the same property functions 
you use in QuickCheck. I do this often for regression testing and to 
ensure that corner cases are covered[1]. Unfortunately this still 
requires a bit of hand-holding to keep your regression suite up to date.

I would *love* there to be a tool which (a) automatically saves failing 
QuickCheck values to disk, and (b) automates using HUnit to load those 
in and test them. I'm not so sure that QuickCheck should be doing the 
second step of that since that'd really mess with the QuickCheck 
infrastructure; once you have the code for reading from disk, it'd be 
trivial to just use HUnit.


[1] I also do some overloading of (==>) and similar, so that I can use 
the same properties in both QuickCheck and lazy SmallCheck. 
Unfortunately, I've yet to come up with a good way to ensure that 
QuickCheck doesn't explore territory already covered by SmallCheck.

-- 
Live well,
~wren