[Haskell-cafe] Re: proposal: HaBench, a Haskell Benchmark Suite

[Al Falloon]

David Roundy wrote:
> On Fri, Jan 26, 2007 at 10:17:28AM -0500, Al Falloon wrote:
>> Kenneth Hoste wrote:
>>> The idea is to gather a bunch of programs written in Haskell, and which 
>>> are representative for the Haskell community (i.e. apps, libraries, 
>>> ...).
>> A While ago I tried to write a Haskell version of John Harrops 
>> ray-tracer benchmark 
>> (http://www.ffconsultancy.com/free/ray_tracer/languages.html) but the 
>> performance was not very good (the OCaml version I based it on was at 
>> least 10x faster).
>>
>> I would be happy to contribute my code to the benchmark suite if you are 
>> interested. Perhaps someone can point out obvious speed-ups that I 
>> missed while trying to improve the performance.
> 
> I would think that what we'd want to benchmark would be clean, optimized
> actually-used code.  I.e. things like Data.Bytestring, so that we could see
> how compilers differed on important code, or how the code generated on
> different architectures differed.  e.g. if jhc beats ghc on amd64, the ghc
> developers would probably be very curious as to why, and how to fix it.
> 
> Code that's not been properly optimized with respect to strictness, etc,
> would fail to focus the tests on important optimizations of the compiler.
> But of course, the benchmark code should also be clean, since we want to
> ensure that our compilers are good enough that we can write useful
> beautiful code that is also fast.

I tried to optimize it, but I couldn't approach the speed of the OCaml 
version. I followed the performance tuning advice from the Wiki, and had 
even resorted to writing the inner loop calculations using all unboxed 
doubles, but without significant improvements. This is exactly the kind 
of code that I write most often, and I would love to see improvements in 
the optimizations for this kind of numerically intensive code 
(especially without having to resort to compiler-specific unboxed 
representations).

I agree that common libraries like ByteString need to be well 
represented, but the original request additionally included programs 
that are representative of applications. A ray-tracer (even with a fixed 
scene and only one type of scene primitive) is a fairly nice 
approximation of a real numerical batch-oriented application while still 
being small enough to understand and modify. I expect thats why John 
chose it as his benchmark application in the first place.

I think it would also be a good idea to include SPJ's web server (I 
think its from an STM paper). A lot of the people outside the Haskell 
community will be able to relate better to metrics like pages/second.