GHC's performance

[Simon Peyton Jones]

Friends

The thread below concerns GHC's performance. I'm writing to ask for your help.

In developing GHC we always run 'validate', which runs a lot of regression tests.  A few of those are performance tests, but because we do so frequently, none of these performance tests run for long, and none depend on other packages.  And they tend to test something very specific.

What we lack is a sustained effort to track
a) The performance of GHC itself
b) The performance of GHC-compiled programs

At GHC HQ we *aspire* to track this stuff, but in practice we simply don't.  Bug-fixing, portability, etc etc always ends up taking priority.

But it's a pity that we don't.  For example, Michal's comment below (that his GHC-compiled program has run 10-20% faster with each release of GHC from 6.12) is fantastic -- but we have no data to back it up, or know whether it's just Michael, or more widely true. I also suspect that sometimes we regress and don't know it. People do report this (e.g. #8971), but it's a bit random.  Again, it would be great to have a more systematic way to know.  In many cases it might be easy to fix; but we can only fix if we know.

We have the nofib suite, and we take that very seriously, but it is showing its age, uses no advanced features, and I'm not sure how representative it is any more.

Johan Tibell and Bryan O'Sullivan agreed to become GHC's Performance Tsars a year or two ago, with a view to focusing on (b) at least, but they are both extremely busy.  And I don't think they even attempt to focus on (a).

So I'm wondering: would anyone like to help here?  It would mean
 * Soliciting and gathering together some more substantial benchmarks
 * Gathering performance numbers
 * Yelling quickly if the numbers go bad.
 * Investigating why (e.g. no one has seriously profiled GHC itself for
   a while, both space and time.  I bet there are improvements to be had
   there.)

Maybe it could be part of the new buildbot team's work?  Certainly it'd make sense to use the same nightly-build infrastructure.

Anyway, I'm advertising that there's an un-met need, and I'd love some help.

Thanks

Simon

| -----Original Message-----
| From: Michal J. Gajda [mailto:mjgajda at gmail.com]
| Sent: 10 April 2014 07:26
| To: ghc-devs at haskell.org; Sergei Trofimovich; Manuel Chakravarty; Simon
| Peyton Jones
| Subject: Re: ghc-7.8-rc2 in -O2 mode eats all stack and RAM on pandoc-
| ,citeproc / highlighting-kate
| 
| Dear Devs,
| 
| On 04/10/2014 09:09 AM, ghc-devs-request at haskell.org wrote:
| > Filed the reproducer as a new ticket:
| >     https://ghc.haskell.org/trac/ghc/ticket/8980
| >
| > [ Looks like highlighting-kate asks to be added to
| >   compiler performance benchmarks (are there such ones?)
| >   It tends to stress ghc all the time:
| >     http://hackage.haskell.org/trac/ghc/ticket/3664
| > ]
| Please consider adding hPDB too, if you want to stress the optimizer.
| It shows GHC optimizer at its best, with at least 10-20% improvement in
| every major version of
| the compiler since 6.12. Unfortunately at cost of very long compile
| times.
| Please let me know if I should submit a driver code for automatic
| benchmarks it (it is in hPDB-examples.) Thanks!
| >> > SpecConstr is too aggressive: it sometimes blows up the program
| badly and we have no good
| solution.  See Trac #7898, #7068, #7944, #5550, #8836.
| And #8960, where GHC runs out of memory. (Only in 7.8.) Should be easy
| to reproduce by just `cabal install hPDB`.
| >> > I notice that the latter three are actually fixed in 7.8, so worth
| trying that.  If it
| still fails, do add instructions to reproduce to one of the above open
| tickets, or make a new one.
| >> >
| >> > Meanwhile you can use -fno-spec-constr to simply switch it off for
| offending modules.  That should get you going.
| --
|   Best regards
|     Michal