[Haskell-cafe] Benchmarking two versions of the same package?

[lucas di cioccio]

You should be able to use Laborantin to organize this kind of experiments.
You'll have to write some boilerplate on how to "build" each package set.
https://hackage.haskell.org/package/laborantin-hs

--Lucas

2015-06-11 12:45 GMT+01:00 Alp Mestanogullari <alpmestan at gmail.com>:

> Oh, Julian Arni just reminded me of Joachim Breitner's Gipeda:
> https://github.com/nomeata/gipeda
>
> This might do the trick, but any suggestion is still welcome!
>
> On Thu, Jun 11, 2015 at 1:41 PM, Alp Mestanogullari <alpmestan at gmail.com>
> wrote:
>
>> Hi -cafe,
>>
>> While we can easily benchmark different functions or libraries easily
>> with criterion, I can't think of a reasonably easy (and accurate!) way of
>> benchmarking two versions of the same package. And not necessarily version
>> as in cabal version -- one of the use cases I have in mind would be running
>> a benchmark suite whenever a PR gets merged to the main branch of a
>> library, so the benchmark would need to compare
>> the performance of the library's-code-before-merging and after.
>>
>> This definitely can't be accomplished with something like criterion
>> because we can't have two different instances of a package in scope for a
>> module, even with -XPackageImports.
>>
>> If we separately build the same program against two instances of the same
>> library and run the benchmarks separately, this might happen far apart
>> enough that the machine running this might be under a different load. This
>> does however seem to be the only actual solution? Run separately at two
>> different commits, diff the numbers, report.
>>
>> I have felt the need for a solution to this for quite some time, but have
>> never needed it bad enough that it became a priority. Thinking about it
>> again today, I thought I should drop an email to the list and see if fellow
>> haskellers have an easy-to-use solution for this? I may very well be
>> overlooking something.
>>
>> --
>> Alp Mestanogullari
>>
>
>
>
> --
> Alp Mestanogullari
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150611/6f70bd24/attachment-0001.html>