[Haskell-cafe] ANNOUNCE: test-framework-golden-1.1

Simon Hengel sol at typeful.net
Mon Oct 8 10:30:53 CEST 2012

On Sun, Oct 07, 2012 at 09:09:07PM +0300, Roman Cheplyaka wrote:
> * Simon Hengel <sol at typeful.net> [2012-10-07 15:45:21+0200]
> > On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:
> > > I can do that indeed, and I guess I could reimplement everything I have
> > > at the moment on top of HUnit.
> > >
> > > However, an important part of functionality isn't there at the moment —
> > > golden file management. You should be able to say, "for this test,
> > > take its current output and write it to the corresponding golden file".
> > >
> > > In order to do that, you need to have access to the list of golden tests
> > > in the suite. This is where implementation details of different test
> > > frameworks start to matter. Probably one can make an abstraction over
> > > test frameworks that would give the list of all golden tests.
> > > (Although when you start abstracting over test frameworks, which are
> > > abstractions themselves, it becomes somewhat funny.)
> >
> > Ok, makes sense.
> >
> > I'm looking forward to give it a try, and see how it compares to using
> > operating system primitives (say `cp') for "golden file management".
> 1. You often want to update not just one test, but all, or some of the
>    tests (when you've made a change and verified that the changes in
>    output are expected). Doing it in command line is certainly possible,
>    but not trivial nor convenient.
> 2. For some tests (like goldenVsString) the output is not captured in a
>    file, so using cp directly is not possible.

Yes and yes.  I have no fixed expectations nor any idea how an "ideal"
interface would look like, but I guess there is room for improvement.
So I'm really looking forward to try it ;)

> > > Speaking of such functionality, correct me if I'm wrong, but neither
> > > HUnit nor hspec won't be able to support it anyway, because they
> > > represent tests as opaque IO actions.
> >
> > It would be easy to extend Hspec to support this in the same way you
> > extend test-framework to support this.  It requires existentials;  the
> > only substantial difference that I can see is that test-framework
> > already uses existentials, while Hspec does not.
> Well, if you are willing to make this change, then I'll try to do my
> part of the job and expose a useful abstraction.

If it gives us something that is useful from a users perspective, I'm
happy to make that change.  I would hope that something like [1] works,

    instance Example GoldenTest where
      evaluateExample c = evaluateExample c . goldenTestToHUnitAssetion
      exampleMetadata   = Just . Metadata

That way the Typeable instance is optional.

Personally, I still think that it may be a good idea to first explore
the design space with test-framework before trying to abstract over it.


[1] https://github.com/sol/hspec/commit/6927f642aea44803b57c2b77548931f6865b0c38

More information about the Haskell-Cafe mailing list