[Haskell-cafe] proposal: HaBench, a Haskell Benchmark Suite

[Andy Georges]

Hi Simon et al,


I've picked up the HaBench/nofib/nobench issue again, needing a decent set of real applications to do some exploring of what people these days call split-compilation. We have a framework that was able to explore GCC optimisations [1] -- the downside there was the dependency of these optimisations on each other, requiring them to be done in certain order -- for a multi-objective search space, and extended this to exploring a JIT compiler [2] for Java in our case -- which posed its own problems. Going one step further, we'd like to  explore the tradeoffs that can be made when compiling on different levels: source to bytecode (in some sense) and bytecode to native. Given that LLVM is quicly becoming a state-of-the-art framework and with the recent GHC support, we figured that Haskell would be an excellent vehicle to conduct our exploration and research (and the fact that some people at our lab have a soft spot for Haskell helps too). Which brings me back to benchmarks.

Are there any inputs available that allow the real part of the suite to run for a sufficiently long time? We're going to use criterion in any case given our own expertise with rigorous benchmarking [3,4], but since we've made a case in the past against short running apps on managed runtime systems [5], we'd love to have stuff that runs at least in the order of seconds, while doing useful things. All pointers are much appreciated.

Or if any of you out there have (recent) apps with inputs that are open source ... let us know.

-- Andy


[1] COLE: Compiler Optimization Level Exploration, Kenneth Hoste and Lieven Eeckhout, CGO 2008
[2] Automated Just-In-Time Compiler Tuning, Kenneth Hoste, Andy Georges and Lieven Eeckhout, CGO 2010
[3] Statistically Rigorous Java Performance Evaluation, Andy Georges, Dries Buytaert and Lieven Eeckhout, OOPSLA 2007
[4] Java Performance Evaluation through Rigorous Replay Compilation, Andy Georges, Lieven Eeckhout and Dries Buytaert, OOPSLA 2008
[5] How Java Programs Interact with Virtual Machines at the Microarchitectural Level, Lieven Eeckhout, Andy Georges, Koen De Bosschere, OOPSLA 2003