[Haskell-cafe] Benchmarking two versions of the same package?

[Alp Mestanogullari]

Hi -cafe,

While we can easily benchmark different functions or libraries easily with
criterion, I can't think of a reasonably easy (and accurate!) way of
benchmarking two versions of the same package. And not necessarily version
as in cabal version -- one of the use cases I have in mind would be running
a benchmark suite whenever a PR gets merged to the main branch of a
library, so the benchmark would need to compare
the performance of the library's-code-before-merging and after.

This definitely can't be accomplished with something like criterion because
we can't have two different instances of a package in scope for a module,
even with -XPackageImports.

If we separately build the same program against two instances of the same
library and run the benchmarks separately, this might happen far apart
enough that the machine running this might be under a different load. This
does however seem to be the only actual solution? Run separately at two
different commits, diff the numbers, report.

I have felt the need for a solution to this for quite some time, but have
never needed it bad enough that it became a priority. Thinking about it
again today, I thought I should drop an email to the list and see if fellow
haskellers have an easy-to-use solution for this? I may very well be
overlooking something.

-- 
Alp Mestanogullari
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150611/b40a73f1/attachment.html>