[Haskell-cafe] Benchmarking two versions of the same package?

[Roman Cheplyaka]

In your CI script, you can rename one of your versions (say, mypkg to
mypkg-new) and then use -XPackageImports.

On 11/06/15 14:41, Alp Mestanogullari wrote:
> Hi -cafe,
> 
> While we can easily benchmark different functions or libraries easily
> with criterion, I can't think of a reasonably easy (and accurate!) way
> of benchmarking two versions of the same package. And not necessarily
> version as in cabal version -- one of the use cases I have in mind would
> be running a benchmark suite whenever a PR gets merged to the main
> branch of a library, so the benchmark would need to compare
> the performance of the library's-code-before-merging and after.
> 
> This definitely can't be accomplished with something like criterion
> because we can't have two different instances of a package in scope for
> a module, even with -XPackageImports.
> 
> If we separately build the same program against two instances of the
> same library and run the benchmarks separately, this might happen far
> apart enough that the machine running this might be under a different
> load. This does however seem to be the only actual solution? Run
> separately at two different commits, diff the numbers, report.
> 
> I have felt the need for a solution to this for quite some time, but
> have never needed it bad enough that it became a priority. Thinking
> about it again today, I thought I should drop an email to the list and
> see if fellow haskellers have an easy-to-use solution for this? I may
> very well be overlooking something.
> 
> -- 
> Alp Mestanogullari
> 
> 
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150611/4acdfa57/attachment.sig>