[Haskell-cafe] Benchmarking two versions of the same package?

Alp Mestanogullari alpmestan at gmail.com
Thu Jun 11 12:22:20 UTC 2015


I thought about this but discarded it quite early on for the hacky nature.
But that definitely looks like my best shot, the more I think about it.
Thanks.

On Thu, Jun 11, 2015 at 1:48 PM, Roman Cheplyaka <roma at ro-che.info> wrote:

> In your CI script, you can rename one of your versions (say, mypkg to
> mypkg-new) and then use -XPackageImports.
>
> On 11/06/15 14:41, Alp Mestanogullari wrote:
> > Hi -cafe,
> >
> > While we can easily benchmark different functions or libraries easily
> > with criterion, I can't think of a reasonably easy (and accurate!) way
> > of benchmarking two versions of the same package. And not necessarily
> > version as in cabal version -- one of the use cases I have in mind would
> > be running a benchmark suite whenever a PR gets merged to the main
> > branch of a library, so the benchmark would need to compare
> > the performance of the library's-code-before-merging and after.
> >
> > This definitely can't be accomplished with something like criterion
> > because we can't have two different instances of a package in scope for
> > a module, even with -XPackageImports.
> >
> > If we separately build the same program against two instances of the
> > same library and run the benchmarks separately, this might happen far
> > apart enough that the machine running this might be under a different
> > load. This does however seem to be the only actual solution? Run
> > separately at two different commits, diff the numbers, report.
> >
> > I have felt the need for a solution to this for quite some time, but
> > have never needed it bad enough that it became a priority. Thinking
> > about it again today, I thought I should drop an email to the list and
> > see if fellow haskellers have an easy-to-use solution for this? I may
> > very well be overlooking something.
> >
> > --
> > Alp Mestanogullari
> >
> >
> > _______________________________________________
> > Haskell-Cafe mailing list
> > Haskell-Cafe at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
> >
>
>
>


-- 
Alp Mestanogullari
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20150611/498e494f/attachment.html>


More information about the Haskell-Cafe mailing list