Measuring compiler performance

[Ben Gamari]

Simon Jakobi via ghc-devs <ghc-devs at haskell.org> writes:

> Hi devs!
>
Hi Simon!

> I've recently started working on a few compiler perf related tickets,
> and wondered how to verify that a change actually has a positive
> impact on compiler performance.
>
> I first looked at the wiki for information on this, but didn't find much:
>
> https://gitlab.haskell.org/ghc/ghc/-/wikis/performance/compiler
> doesn't contain any information on how to measure compiler performance
>
> The best tip I found was the recommendation in
> https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-no-fib to
> compile nofib/spectral/simple/Main.hs. With -O0 and -O that takes
> respectively about 1.5s and 5s for me, so the effort is manageable.
> I've also done full nofib runs, but they take a very long time.
>
As Joachim points out, progress on this proceeds in fits and starts.
I have a number of pointers to share:

 * When performing compiler measurements myself I generally use a
   combination of the following:

    * nofib
    * the GHC performance testsuite
    * what I call the "Cabal test"; namely:

        $ _build/stage1/bin/ghc -O -ilibraries/Cabal/Cabal \
          libraries/Cabal/Cabal/Setup.hs +RTS -s

 * My WIP nofib branch [1] makes nofib much faster and easier to work
   with and adds the ability to measure perf counters, in addition to
   the usual RTS and cachegrind statistics.

 * My nofib branch produces output in a uniform, easy to consume format
   and provides a tool for comparing sets of measurements in this format.

 * My ghc_perf tool [2] is very useful for extracting runtime and perf
   statistics from Haskell program runs; furthermore, it produces output
   in the same format as expected by the aforementioned nofib-compare
   utility.

 * I have a utility [3] which I use to reproducibly build a set of
   branches, run the testsuite, nofib, and the Cabal test on each of
   them. Admittedly it could use a bit of cleanup but it does its job
   reasonably well, making performance measurement a "set it and forget
   it" sort of task.

 * We collect and record a complete set of testsuite statistics (saved
   to git notes 43]); however, we currently do not import these into
   gipeda.

 * We don't currently have a box which can measure reliable timings
   (since our builders are nearly all virtualised cloud instances). I'm
   going to need to do some shuffling to change this.

 * One potentially useful source of performance information (which sadly
   we currently do not exploit) is the -ddump-timing output produced
   during head.hackage runs.

[1] https://gitlab.haskell.org/ghc/nofib/merge_requests/24
[2] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py
[3] https://gitlab.haskell.org/bgamari/ghc-utils/-/tree/master/build-all
[4] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests


> A problem in this context is that reliable performance measurements
> require a quiet machine. Closing my browser, and turning off other
> programs is – in my perception – rather inconvenient, particularly
> when I have to do it for a prolonged time.
>
> Ideally I wouldn't have to perform these measurements on my local
> machine at all! Do you usually use a separate machine for this? _Very_
> convenient would be some kind of bot whom I could tell e.g.
>

Indeed it is inconvenient. I am in the lucky situation that I have
another machine locally that can be made reasonably quiet without
interfering with my worflow. However, in general

> @perf-bot compiler perf
>
> …or more concretely
>
> @perf-bot compile nofib/spectral/simple/Main.hs
>
> …or just
>
> @nofib-bot run
>
> … or something like that.
>
> I've noticed that CI now includes a perf-nofib job. But since it
> appears to run on a different machine each time, I'm not sure whether
> it's actually useful for comparing performance. Could it be made more
> useful by running it consistently on the same dedicated machine?
>
Indeed, we currently don't have a dedicated machine for timings.
However, allocations and executable sizes are still useful.

Nevertheless, as noted above I think that we should make more of an
effort to measure time. I need to do some shuffling of our runners so we
have a quiet bare-metal which can be dedicated to performance
measurement. I'll try to get to this in the next day or so.

> Another question regarding performing compiler perf measurements
> locally is which build flavour to use: So far I have used the "perf"
> flavour. A problem here is that a full build seems to take close to an
> hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> this the right flavour to use?
>
I think perf is the best option for performance measurement (afterall,
we want to know what users would see). However, it is indeed a bit
painful.

> BTW what's the purpose of the profiled GHC modules built with this
> flavour which just seem to additionally prolong compile time? I don't
> see a ghc-prof binary or similar in _build/stage1/bin.
>
Indeed; there is little sense in building profiled modules just for
performance measurement. However, I don't believe we currently have a
build flavour which provides comparable optimisation but without the
profiled way. Perhaps we should add one.

> Also, what's the status of gipeda? The most recent commit at
> https://perf.haskell.org/ghc/ is from "about a year ago"?
>
Indeed the machine which was previously providing gipeda builds is sadly
no longer around; consequently it's on ice at the moment. I would like
to get it going again but recently correctness issues have been taking
up more time than I would like to admit.

> Sorry for this load of questions and complaints! I do believe though
> that if work on compiler performance was a bit better documented and
> more convenient, we might see even more progress on that front. :)
>
Quite alright! Typing out the points above made me realize that there is
indeed quite a bit of knowledge that the wiki leaves un-said.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200406/ebc247cf/attachment.sig>