nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2

[Austin Seipp]

On Tue, Feb 5, 2013 at 2:54 AM, Simon Peyton-Jones
<simonpj at microsoft.com> wrote:
> Austin, a 25% performance regression moving to 7.6 is not AT ALL what I expect. I generally expect modest performance improvements?  Can you characterise more precisely what is happening?  The place I always start is to compile the entire thing with -ticky and see where allocation is changing.  (Using -prof affects the optimiser too much.)

I have CC'd Eyal just in case. The discussion was informal but he can
hopefully provide more context and rigor. I think off hand, this
occurred in a rather large-ish application of his (Lamdu?,) and so
tracking down precise reasons may prove difficult. I think the most
likely case is just those few 'small cuts' accumulate quickly and are
reflecting poorly for this particular case - and that's really the
worse 'bug report' of all!

Hashable/lens alone for example could certainly make a sizable impact
here when added up, e.g. [1] is a recent example of an alleged perf
anomaly as of late. And the OS could certainly be relevant.[2] All the
more reason to expand nofib and get those bots up!

[1] https://github.com/tibbe/hashable/issues/57

[2] Just thinking out loud, but, whenever this happens we really need
to characterize results on a per-OS/hardware basis if possible in the
future, with some relatively detailed hardware info, to be
unambiguous. In terms of raw CPU speed, a lot of benchmarks probably
won't stand out due to the OS. But OS X is scheduled to get worse in
the SMP case soon[3] for example, and if we inevitably try and start
doing things like latency or I/O benchmarks, I'm more than certain
things will pop up here.

[3] See this ticket: http://hackage.haskell.org/trac/ghc/ticket/7602

> Simon
>
> | -----Original Message-----
> | From: ghc-devs-bounces at haskell.org [mailto:ghc-devs-bounces at haskell.org]
> | On Behalf Of Austin Seipp
> | Sent: 05 February 2013 04:22
> | To: Johan Tibell
> | Cc: ghc-devs at haskell.org
> | Subject: Re: nofib comparisons between 7.0.4, 7.4.2, 7.6.1, and 7.6.2
> |
> | I'm +1 for this. Eyal Lotem and I were just discussing this on IRC a few
> | minutes ago, and he suffered a rather large (~25%) performance hit when
> | upgrading to 7.6.1, which is unfortunate.
> |
> | Committers are typically very good about recording nofib results in
> | their commit and being performance-courteous, but I'm not sure there's
> | ever been a longer-scale view of GHC performance over multiple releases
> | like this - or even a few months. At least not recently. On top of that,
> | his application was a type checker, which may certainly stress different
> | performance points than what nofib might. Once we get performance bots
> | set up, I've got a small set of machines I'm willing to throw at it.
> |
> | Thanks for the results, Johan!
> |
> | On Mon, Feb 4, 2013 at 4:33 PM, Johan Tibell <johan.tibell at gmail.com>
> | wrote:
> | > Hi all,
> | >
> | > I haven't had much time to do performance tzar work yet, but I did run
> | > nofib on the last few GHC releases to see the current trend. The
> | > benchmarks where run on my 64-bit Core i7-3770 @ 3.40GHz Linux
> | machine. Here are the results:
> | >
> | > 7.0.4 to 7.4.2:
> | >
> | > ----------------------------------------------------------------------
> | ----------
> | >         Program           Size    Allocs   Runtime   Elapsed  TotalMem
> | > ----------------------------------------------------------------------
> | ----------
> | >             Min          -1.6%    -57.3%    -39.1%    -36.4%    -25.0%
> | >             Max         +21.5%   +121.5%    +24.5%    +25.4%   +300.0%
> | >  Geometric Mean          +8.5%     -0.7%     -7.1%     -5.2%     +2.0%
> | >
> | > The big loser here in terms of runtime is "kahan", which I added to
> | > test tight loops involving unboxed arrays and floating point
> | > arithmetic. I believe there was a regression in fromIntegral RULES
> | > during this release, which meant that some conversions between
> | > fixed-width types went via Integer, causing unnecessary allocation.
> | >
> | > 7.4.2 to 7.6.1:
> | >
> | > ----------------------------------------------------------------------
> | ----------
> | >         Program           Size    Allocs   Runtime   Elapsed  TotalMem
> | > ----------------------------------------------------------------------
> | ----------
> | >             Min          -5.1%    -23.8%    -11.8%    -12.9%    -50.0%
> | >             Max          +5.3%   +225.5%     +7.2%     +8.8%   +200.0%
> | >  Geometric Mean          -0.4%     +2.1%     +0.3%     +0.2%     +0.7%
> | >
> | > The biggest loser here in terms of runtime is "integrate". I haven't
> | > looked into why yet.
> | >
> | > 7.6.1 to 7.6.2:
> | >
> | > ----------------------------------------------------------------------
> | ----------
> | >         Program           Size    Allocs   Runtime   Elapsed  TotalMem
> | > ----------------------------------------------------------------------
> | ----------
> | >             Min          -2.9%     +0.0%     -4.8%     -4.4%     -1.9%
> | >             Max          +0.0%     +1.0%     +4.5%     +6.4%    +20.8%
> | >  Geometric Mean          -1.7%     +0.0%     +0.1%     +0.3%     +0.2%
> | >
> | > I have two takeaways:
> | >
> | >  * It's worthwhile running nofib before releases as it does find some
> | > programs that regressed.
> | >  * There are some other regressions out there (i.e. in code on
> | > Hackage) that aren't reflected here, suggesting that we need to add
> | > more programs to nofib.
> | >
> | > Cheers,
> | > Johan
> | >
> | >
> | > _______________________________________________
> | > ghc-devs mailing list
> | > ghc-devs at haskell.org
> | > http://www.haskell.org/mailman/listinfo/ghc-devs
> | >
> |
> |
> |
> | --
> | Regards,
> | Austin
> |
> | _______________________________________________
> | ghc-devs mailing list
> | ghc-devs at haskell.org
> | http://www.haskell.org/mailman/listinfo/ghc-devs



-- 
Regards,
Austin