[Haskell-cafe] Re: [Haskell] ANNOUNCE: The Fibon benchmark suite (v0.2.0)

[David Peixotto]

On Nov 9, 2010, at 3:45 PM, Jason Dagit wrote:
> I have a few questions:
>   * What differentiates fibon from criterion?  I see both use the statistics package.

I think the two packages have different benchmarking targets.

Criterion allows you to easily test individual functions and gives some help with benchmarking in the presence of lazy evaluation. If some code does not execute for a long time it will run it multiple times to get sensible timings. Criterion does a much more sophisticated statistical analysis of the results, but I hope to incorporate that into the Fibon analysis in the future.

Fibon is a more traditional benchmarking suite like SPEC or nofib. My interest is using it to test compiler optimizations. It can only benchmark at the whole program level by running an executable. It checks that the program produces the correct output, can collect extra metrics generated by the program, separates collecting results from analyzing results, and generates tables directly comparing the results from different benchmark runs.

>   * Does it track memory statistics?  I glanced at the FAQ but didn't see anything about it.

Yes, it can read memory statistics dumped by the GHC runtime. It has built in support for reading the stats dumped by `+RTS -t --machine-readable` which includes things like bytes allocated and time spent in GC.

>   * Are the numbers in the sample output seconds or milliseconds?  What is the stddev (eg., what does the distribution of run-times look like)?

I'm not sure which results you are referring to exactly (the numbers in the announcement were lines of code). I picked benchmarks that all ran for at least a second (and hopefully longer) with compiler optimizations enabled. On an 8-core Xeon, the median time over all benchmarks is 8.43 seconds, mean time is 12.57 seconds and standard deviation is 14.56 seconds.

-David

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org/pipermail/haskell-cafe/attachments/20101109/ffbadd2f/attachment.html