Measuring compiler performance

[Richard Eisenberg]

Because of the fragility of time-based performance tests, I often use allocations as a proxy for time. The number of allocations is very stable. On the down side, it is sometimes possible to reduce allocations while increasing time, so one shouldn't try too hard to blindly optimize allocations.

Separately, in order to get measurements, I build Cabal with ghc --make. Cabal is convenient because it is sizeable, real-world, updated, and needs no dependencies. Compiling the full library definitely takes more than 5 seconds, though.

I agree that there is plenty of room for improvement here!

Richard

> On Apr 5, 2020, at 12:24 AM, Simon Jakobi via ghc-devs <ghc-devs at haskell.org> wrote:
> 
> Hi devs!
> 
> I've recently started working on a few compiler perf related tickets,
> and wondered how to verify that a change actually has a positive
> impact on compiler performance.
> 
> I first looked at the wiki for information on this, but didn't find much:
> 
> https://gitlab.haskell.org/ghc/ghc/-/wikis/performance/compiler
> doesn't contain any information on how to measure compiler performance
> 
> The best tip I found was the recommendation in
> https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-no-fib to
> compile nofib/spectral/simple/Main.hs. With -O0 and -O that takes
> respectively about 1.5s and 5s for me, so the effort is manageable.
> I've also done full nofib runs, but they take a very long time.
> 
> A problem in this context is that reliable performance measurements
> require a quiet machine. Closing my browser, and turning off other
> programs is – in my perception – rather inconvenient, particularly
> when I have to do it for a prolonged time.
> 
> Ideally I wouldn't have to perform these measurements on my local
> machine at all! Do you usually use a separate machine for this? _Very_
> convenient would be some kind of bot whom I could tell e.g.
> 
> @perf-bot compiler perf
> 
> …or more concretely
> 
> @perf-bot compile nofib/spectral/simple/Main.hs
> 
> …or just
> 
> @nofib-bot run
> 
> … or something like that.
> 
> I've noticed that CI now includes a perf-nofib job. But since it
> appears to run on a different machine each time, I'm not sure whether
> it's actually useful for comparing performance. Could it be made more
> useful by running it consistently on the same dedicated machine?
> 
> Another question regarding performing compiler perf measurements
> locally is which build flavour to use: So far I have used the "perf"
> flavour. A problem here is that a full build seems to take close to an
> hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> this the right flavour to use?
> 
> BTW what's the purpose of the profiled GHC modules built with this
> flavour which just seem to additionally prolong compile time? I don't
> see a ghc-prof binary or similar in _build/stage1/bin.
> 
> Also, what's the status of gipeda? The most recent commit at
> https://perf.haskell.org/ghc/ is from "about a year ago"?
> 
> Sorry for this load of questions and complaints! I do believe though
> that if work on compiler performance was a bit better documented and
> more convenient, we might see even more progress on that front. :)
> 
> Cheers,
> Simon
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs