nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

[Simon Peyton-Jones]

I'm +10.  This is precisely the reason we have our supreme Performance Tsars, to keep us honest.  GHC leadership is become increasingly decentralised and I am truly grateful to Bryan and Johan for picking up this particular challenge.

My guess is that regressions are accidental and readily fixed, but we can't fix them if we don't know about them.

Johan mentions more nofib benchmarks: yes please!  But someone has to put them in.

Austin, a 25% performance regression moving to 7.6 is not AT ALL what I expect. I generally expect modest performance improvements?  Can you characterise more precisely what is happening?  The place I always start is to compile the entire thing with -ticky and see where allocation is changing.  (Using -prof affects the optimiser too much.)

Simon

| -----Original Message-----
| From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
| On Behalf Of Austin Seipp
| Sent: 05 February 2013 04:22
| To: Johan Tibell
| Cc: ghc-devs at haskell.org
| Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2
| 
| I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a few
| minutes ago, and he suffered a rather large (~25%) performance hit when
| upgrading to 7.6.1, which is unfortunate.
| 
| Committers are typically very good about recording nofib results in
| their commit and being performance-courteous, but I'm not sure there's
| ever been a longer-scale view of GHC performance over multiple releases
| like this - or even a few months. At least not recently. On top of that,
| his application was a type checker, which may certainly stress different
| performance points than what nofib might. Once we get performance bots
| set up, I've got a small set of machines I'm willing to throw at it.
| 
| Thanks for the results, Johan!
| 
| On Mon, Feb 4, 2013 at 4:33 PM, Johan Tibell <johan.tibell at gmail.com>
| wrote:
| > Hi all,
| >
| > I haven't had much time to do performance tzar work yet, but I did run
| > nofib on the last few GHC releases to see the current trend. The
| > benchmarks where run on my 64-bit Core i7-3770 @ 3.40GHz Linux
| machine. Here are the results:
| >
| > 7.0.4 to 7.4.2:
| >
| > ----------------------------------------------------------------------
| ----------
| >         Program           Size    Allocs   Runtime   Elapsed  TotalMem
| > ----------------------------------------------------------------------
| ----------
| >             Min          -1.6%    -57.3%    -39.1%    -36.4%    -25.0%
| >             Max         +21.5%   +121.5%    +24.5%    +25.4%   +300.0%
| >  Geometric Mean          +8.5%     -0.7%     -7.1%     -5.2%     +2.0%
| >
| > The big loser here in terms of runtime is "kahan", which I added to
| > test tight loops involving unboxed arrays and floating point
| > arithmetic. I believe there was a regression in fromIntegral RULES
| > during this release, which meant that some conversions between
| > fixed-width types went via Integer, causing unnecessary allocation.
| >
| > 7.4.2 to 7.6.1:
| >
| > ----------------------------------------------------------------------
| ----------
| >         Program           Size    Allocs   Runtime   Elapsed  TotalMem
| > ----------------------------------------------------------------------
| ----------
| >             Min          -5.1%    -23.8%    -11.8%    -12.9%    -50.0%
| >             Max          +5.3%   +225.5%     +7.2%     +8.8%   +200.0%
| >  Geometric Mean          -0.4%     +2.1%     +0.3%     +0.2%     +0.7%
| >
| > The biggest loser here in terms of runtime is "integrate". I haven't
| > looked into why yet.
| >
| > 7.6.1 to 7.6.2:
| >
| > ----------------------------------------------------------------------
| ----------
| >         Program           Size    Allocs   Runtime   Elapsed  TotalMem
| > ----------------------------------------------------------------------
| ----------
| >             Min          -2.9%     +0.0%     -4.8%     -4.4%     -1.9%
| >             Max          +0.0%     +1.0%     +4.5%     +6.4%    +20.8%
| >  Geometric Mean          -1.7%     +0.0%     +0.1%     +0.3%     +0.2%
| >
| > I have two takeaways:
| >
| >  * It's worthwhile running nofib before releases as it does find some
| > programs that regressed.
| >  * There are some other regressions out there (i.e. in code on
| > Hackage) that aren't reflected here, suggesting that we need to add
| > more programs to nofib.
| >
| > Cheers,
| > Johan
| >
| >
| > _______________________________________________
| > ghc-devs mailing list
| > ghc-devs at haskell.org
| > http://www.haskell.org/mailman/listinfo/ghc-devs
| >
| 
| 
| 
| --
| Regards,
| Austin
| 
| _______________________________________________
| ghc-devs mailing list
| ghc-devs at haskell.org
| http://www.haskell.org/mailman/listinfo/ghc-devs