[Haskell-cafe] Problems using tasty-golden

[Roman Cheplyaka]

* Jan Stolarek <jan.stolarek at p.lodz.pl> [2014-02-14 14:59:31+0100]
> Hi *,
> 
> I've been trying to write a testsuite using tasty-golden and I've run into two technical problems. 
> 
> I'll start with some background. I'm working on singletons library that provides Template Haskell 
> functionality. I would like to tests whether some particular TH splices compile at all and if 
> they compile whether they produce expected output. My plan is to attempt compilation of a file 
> with GHC. If compilation fails then the test fails. If it succeedes then output is compared with 
> a golden file. I am directly calling goldenTest function from Test.Tasty.Golden.Internal. I wrote 
> this function to produce the test output:
> 
> compileWithGHC :: IO (Either BSL.ByteString BSL.ByteString)
> 
> It runs GHC with some specified options and returns Left stderr (if compilation fails) or Right 
> stdout (if it succeedes). This function is passed as third parameter to goldenTest (using 
> liftIO). Then I have a comparing function:
> 
> cmp :: Either BSL.ByteString BSL.ByteString 
>      -> Either BSL.ByteString BSL.ByteString 
>      -> IO (Maybe String)
> 
> If it gets two Rights then it compares them. If the second param is left then the function returns 
> Just stderr.
> 
> I've run into following problems:
> 
> 1. When I try to run my test I get:
> 
> message threw an exception: MY_GOLDEN_FILE: hGetBufSome: illegal operation (handle is closed)
> 
> I suspect that lazy IO is getting in the way but I have no idea why exactly things go wrong.

That's a plausible explanations. File handles are closed when the
ValueGetter monad is run. Try to add something like

  Control.Exception.evaluate $ Control.DeepSeq.force cmpResult

to your comparison function. (I think I should build this forcing
behaviour right into tasty-golden itself.)

(Filed as https://github.com/feuerbach/tasty-golden/issues/7)

> 2. I had to hardcode the path to my golden file. Otherwise the file could not be found. I suspect 
> the reason is that test are run somewhere in dist/ subdirectory whereas my golden files are 
> located alongside my test files. What can I do to have the golden files located without giving 
> specific subdirectories in the source code?

Golden files are looked up in an ordinary way — i.e. relative paths are
interpreted relative to the test program's CWD. So, for example, if your
test files are under 'test/' subdirectory of your project's main directory,
and you want to run the test suite from your project's main directory,
then your paths must look like "test/...".

(Filed as https://github.com/feuerbach/tasty-golden/issues/8)

> A bonus question. I redefined vgReadFile from Test.Tasty.Golden.Internal to return Either:
> 
> vgReadFileE :: FilePath -> ValueGetter r (Either BSL.ByteString BSL.ByteString)
> vgReadFileE path =
>   (fmap Right . liftIO . BSL.hGetContents =<<) $
>   ValueGetter $
>   ContT $ \k ->
>   bracket
>     (openBinaryFile path ReadMode)
>     hClose
>     k
> 
> The only modification is the "fmap Right .". How can I return Left if opening the file fails?

First off, you don't have to do that (unless you have some special needs
— if so, what are they?). If the file doesn't exist, tasty will catch
the exception and handle it gracefully.

If you really want to do it (or are just curious), the easiest way is to
revert your addition of 'fmap Right' and replace the 'bracket' code with
'try' code.

Roman
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: Digital signature
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20140214/3d708363/attachment.sig>