GHC's performance

Fri Apr 11 09:58:02 UTC 2014

On 10/04/2014 08:58, Simon Peyton Jones wrote:
> Friends
>
> The thread below concerns GHC's performance. I'm writing to ask for your help.
>
> In developing GHC we always run 'validate', which runs a lot of regression tests.  A few of those are performance tests, but because we do so frequently, none of these performance tests run for long, and none depend on other packages.  And they tend to test something very specific.
>
> What we lack is a sustained effort to track
> a) The performance of GHC itself
> b) The performance of GHC-compiled programs
>
> At GHC HQ we *aspire* to track this stuff, but in practice we simply don't.  Bug-fixing, portability, etc etc always ends up taking priority.
>
> But it's a pity that we don't.  For example, Michal's comment below (that his GHC-compiled program has run 10-20% faster with each release of GHC from 6.12) is fantastic -- but we have no data to back it up, or know whether it's just Michael, or more widely true. I also suspect that sometimes we regress and don't know it. People do report this (e.g. #8971), but it's a bit random.  Again, it would be great to have a more systematic way to know.  In many cases it might be easy to fix; but we can only fix if we know.
>
> We have the nofib suite, and we take that very seriously, but it is showing its age, uses no advanced features, and I'm not sure how representative it is any more.
>
> Johan Tibell and Bryan O'Sullivan agreed to become GHC's Performance Tsars a year or two ago, with a view to focusing on (b) at least, but they are both extremely busy.  And I don't think they even attempt to focus on (a).
>
> So I'm wondering: would anyone like to help here?  It would mean
>   * Soliciting and gathering together some more substantial benchmarks
>   * Gathering performance numbers
>   * Yelling quickly if the numbers go bad.
>   * Investigating why (e.g. no one has seriously profiled GHC itself for
>     a while, both space and time.  I bet there are improvements to be had
>     there.)
>
> Maybe it could be part of the new buildbot team's work?  Certainly it'd make sense to use the same nightly-build infrastructure.
>
> Anyway, I'm advertising that there's an un-met need, and I'd love some help.

Let me second this.  In particular, I think we regress on GHC 
performance regularly, because the perf tests just aren't a good way to 
prevent regressions.  When we have a +/- 10% window, someone can commit 
a 9% regression without triggering a perf failure, but the next patch to 
come along with a 1% regression will be unfairly blamed.

Furthermore, by the time we get to the perf tests we're nearly done and 
just want to push and go home, not go back and profile GHC.  Yet the 
perf tests have an important purpose: the idea is to catch the problem 
when we have the crucial piece of information: the patch that caused the 
regression.  Someone can try to optimise GHC later, but they have to 
start from scratch without the information about what caused the 
regressions.

Having an automated system to track GHC performance would help a lot 
with this, I think.

Cheers,
Simon

>
> Thanks
>
> Simon
>
> | -----Original Message-----
> | From: Michal J. Gajda [mailto:mjgajda at gmail.com]
> | Sent: 10 April 2014 07:26
> | To: ghc-devs at haskell.org; Sergei Trofimovich; Manuel Chakravarty; Simon
> | Peyton Jones
> | Subject: Re: ghc-7.8-rc2 in -O2 mode eats all stack and RAM on pandoc-
> | ,citeproc / highlighting-kate
> |
> | Dear Devs,
> |
> | On 04/10/2014 09:09 AM, ghc-devs-request at haskell.org wrote:
> | > Filed the reproducer as a new ticket:
> | >     https://ghc.haskell.org/trac/ghc/ticket/8980
> | >
> | > [ Looks like highlighting-kate asks to be added to
> | >   compiler performance benchmarks (are there such ones?)
> | >   It tends to stress ghc all the time:
> | >     http://hackage.haskell.org/trac/ghc/ticket/3664
> | > ]
> | Please consider adding hPDB too, if you want to stress the optimizer.
> | It shows GHC optimizer at its best, with at least 10-20% improvement in
> | every major version of
> | the compiler since 6.12. Unfortunately at cost of very long compile
> | times.
> | Please let me know if I should submit a driver code for automatic
> | benchmarks it (it is in hPDB-examples.) Thanks!
> | >> > SpecConstr is too aggressive: it sometimes blows up the program
> | badly and we have no good
> | solution.  See Trac #7898, #7068, #7944, #5550, #8836.
> | And #8960, where GHC runs out of memory. (Only in 7.8.) Should be easy
> | to reproduce by just `cabal install hPDB`.
> | >> > I notice that the latter three are actually fixed in 7.8, so worth
> | trying that.  If it
> | still fails, do add instructions to reproduce to one of the above open
> | tickets, or make a new one.
> | >> >
> | >> > Meanwhile you can use -fno-spec-constr to simply switch it off for
> | offending modules.  That should get you going.
> | --
> |   Best regards
> |     Michal
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://www.haskell.org/mailman/listinfo/ghc-devs
>