Measuring compiler performance

[Simon Jakobi]

Hi devs!

I've recently started working on a few compiler perf related tickets,
and wondered how to verify that a change actually has a positive
impact on compiler performance.

I first looked at the wiki for information on this, but didn't find much:

https://gitlab.haskell.org/ghc/ghc/-/wikis/performance/compiler
doesn't contain any information on how to measure compiler performance

The best tip I found was the recommendation in
https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-no-fib to
compile nofib/spectral/simple/Main.hs. With -O0 and -O that takes
respectively about 1.5s and 5s for me, so the effort is manageable.
I've also done full nofib runs, but they take a very long time.

A problem in this context is that reliable performance measurements
require a quiet machine. Closing my browser, and turning off other
programs is – in my perception – rather inconvenient, particularly
when I have to do it for a prolonged time.

Ideally I wouldn't have to perform these measurements on my local
machine at all! Do you usually use a separate machine for this? _Very_
convenient would be some kind of bot whom I could tell e.g.

@perf-bot compiler perf

…or more concretely

@perf-bot compile nofib/spectral/simple/Main.hs

…or just

@nofib-bot run

… or something like that.

I've noticed that CI now includes a perf-nofib job. But since it
appears to run on a different machine each time, I'm not sure whether
it's actually useful for comparing performance. Could it be made more
useful by running it consistently on the same dedicated machine?

Another question regarding performing compiler perf measurements
locally is which build flavour to use: So far I have used the "perf"
flavour. A problem here is that a full build seems to take close to an
hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
this the right flavour to use?

BTW what's the purpose of the profiled GHC modules built with this
flavour which just seem to additionally prolong compile time? I don't
see a ghc-prof binary or similar in _build/stage1/bin.

Also, what's the status of gipeda? The most recent commit at
https://perf.haskell.org/ghc/ is from "about a year ago"?

Sorry for this load of questions and complaints! I do believe though
that if work on compiler performance was a bit better documented and
more convenient, we might see even more progress on that front. :)

Cheers,
Simon