[Haskell-cafe] How to correctly benchmark code with Criterion?

[Janek S.]

> Something like this might work, not sure what the canonical way is.
> (...)

This is basically the same as the answer I was given on SO. My concerns about this solutions are:
- rnf requires its parameter to belong to NFData type class. This is not the case for some data 
structures like Repa arrays.
- evaluate only evaluates its argument to WHNF - is this enough? If I have a tuple containing two 
lists won't this only evaluate the tuple construtor and leave the lists as thunks? This is 
actually the case in my code.

As I said previously, it seems that Criterion somehow evaluates the data so that time needed for 
its creation is not included in the benchmark. I modified my dataBuild function to look lik this:

dataBuild gen = unsafePerformIO $ do
    let x = (take 6 $ randoms gen, take 2048 $ randoms gen)
    delayThread 1000000
    return x

When I ran the benchmark, criterion estimated the time needed to complete it to over 100 seconds 
(which means that delayThread worked and was used as a basis for estimation), but the benchamrk 
was finished much faster and there was no difference in the final result comparing to the normal 
dataBuild function. This suggests that once data was created and used for estimation, the 
dataBuild function was not used again. The main question is: is this observation correct? In this 
question on SO: 
http://stackoverflow.com/questions/6637968/how-to-use-criterion-to-measure-performance-of-haskell-programs 
one of the aswers says that there is no automatic memoization, while it looks that in fact the 
values of dataBuild are memoized. I have a feeling that I am misunderstanding something.

> I don't know if you have already read them,
> but Tibell's slides on High Performance Haskell are pretty good:
>
> http://www.slideshare.net/tibbe/highperformance-haskell
>
> There is a section at the end where he runs several tests using Criterion.
I skimmed the slides and slide 59 seems to show that my concerns regarding WHNF might be true.

Janek