[Haskell-cafe] Benchmarking harnesses for a more modern nofib?

[Joachim Breitner]

Hi,

definitely interested. I have just talked about this with Richard
yesterday, and we quite agree with what you observed. As the maintainer
of http://perf.haskell.org/ghc, I’d very much welcome better data!

Note that fibon already has bitrotted, and does not quite work any
more. So there is some low hanging fruit in resurrecting that one.

Simon mentioned a few points, such as dependencies. But note that you
can relatively easily dump the dependencies’s modules in your source
repository to both bundle and freeze them. I’d prefer that to (even
strict) dependencies to something external, as that lowers the barrier
for developers to actually run the benchmarks!

Another important step in that direction would be to define a common
output for benchmark suites defined in .cabal files, so it is easier to
set up things like http://perf.haskell.org/ghc and http://perf.haskell.
org/binary for these projects.

About the harness: haskell.org is currently paying a student (CCed) to
setup a travis-like infrastructure based on gipeda (the software behind
perf.haskell.org) that would allow library authors to very simply get
continuous benchmark measurements. Let’s see what comes out of that!

Greetings,
Joachim

Am Montag, den 04.04.2016, 01:06 -0400 schrieb Ryan Newton:
> Hi all,
> 
> Is anyone currently working in, or interested in helping with, a new
> benchmark suite for Haskell?  Perhaps, packaging up existing apps and
> app benchmarks into a new benchmark suite that gives a broad picture
> of Haskell application performance today?
> 
> Background: We run nofib, and we run the shootout benchmarks.  But
> when we want to evaluate basic changes to GHC optimizations or data
> representation, these really don't give us a full picture of whether
> a change is beneficial.
> 
> A few years ago, fibon tried to gather some Hackage benchmarks.  This
> may work even better with Stackage, where there are 180 benchmark
> suites among the 1770 packages currently.
> 
> Also, these days companies are building substantial apps in Hackage. 
> Which substantial apps could or should go into a benchmark suite?  I
> see Warp and other web server benchmarks all over the web.  But is
> there a harness that can time some of this code while running inside
> a single-machine, easy-setup benchmark suite?
> 
> Best,
>   -Ryan
> 
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
-- 
Joachim “nomeata” Breitner
  mail at joachim-breitner.de • https://www.joachim-breitner.de/
  XMPP: nomeata at joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F
  Debian Developer: nomeata at debian.org

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20160404/e4fbfbd6/attachment.sig>