Measuring compiler performance

[Matthew Pickering]

Simon, I assume the `-no-link` flag does this.

```
> ghc --show-options | grep link
-copy-libs-when-linking
-no-link
-no-auto-link-packages
--print-c-compiler-link-flags
```

Cheers,

Matt

On Wed, Apr 8, 2020 at 4:15 PM Simon Jakobi via ghc-devs
<ghc-devs at haskell.org> wrote:
>
> Many thanks, Richard, Andreas, Joachim, and Ben, for your responses! I
> have a few things to try now. :)
>
> >     * what I call the "Cabal test"; namely:
> >
> >         $ _build/stage1/bin/ghc -O -ilibraries/Cabal/Cabal \
> >           libraries/Cabal/Cabal/Setup.hs +RTS -s
>
> Thanks for spelling it out like that, Ben! I'm slightly embarrassed to
> say that I hadn't been aware that I could use GHC directly in this way
> to build a package!
>
> Andreas, you wrote:
>
> > In general I only compile as linking adds overhead which isn't really part of GHC.
>
> How do I tell GHC to build e.g. nofib/spectral/simple/Main.hs or Cabal
> without linking?
>
> I'll eventually try to distill a wiki page from all this!
>
> Cheers,
> Simon
>
>
>
>
>
> >
> >  * My WIP nofib branch [1] makes nofib much faster and easier to work
> >    with and adds the ability to measure perf counters, in addition to
> >    the usual RTS and cachegrind statistics.
> >
> >  * My nofib branch produces output in a uniform, easy to consume format
> >    and provides a tool for comparing sets of measurements in this format.
> >
> >  * My ghc_perf tool [2] is very useful for extracting runtime and perf
> >    statistics from Haskell program runs; furthermore, it produces output
> >    in the same format as expected by the aforementioned nofib-compare
> >    utility.
> >
> >  * I have a utility [3] which I use to reproducibly build a set of
> >    branches, run the testsuite, nofib, and the Cabal test on each of
> >    them. Admittedly it could use a bit of cleanup but it does its job
> >    reasonably well, making performance measurement a "set it and forget
> >    it" sort of task.
> >
> >  * We collect and record a complete set of testsuite statistics (saved
> >    to git notes 43]); however, we currently do not import these into
> >    gipeda.
> >
> >  * We don't currently have a box which can measure reliable timings
> >    (since our builders are nearly all virtualised cloud instances). I'm
> >    going to need to do some shuffling to change this.
> >
> >  * One potentially useful source of performance information (which sadly
> >    we currently do not exploit) is the -ddump-timing output produced
> >    during head.hackage runs.
> >
> > [1] https://gitlab.haskell.org/ghc/nofib/merge_requests/24
> > [2] https://gitlab.haskell.org/bgamari/ghc-utils/blob/master/ghc_perf.py
> > [3] https://gitlab.haskell.org/bgamari/ghc-utils/-/tree/master/build-all
> > [4] https://gitlab.haskell.org/ghc/ghc/-/wikis/building/running-tests/performance-tests
> >
> >
> > > A problem in this context is that reliable performance measurements
> > > require a quiet machine. Closing my browser, and turning off other
> > > programs is – in my perception – rather inconvenient, particularly
> > > when I have to do it for a prolonged time.
> > >
> > > Ideally I wouldn't have to perform these measurements on my local
> > > machine at all! Do you usually use a separate machine for this? _Very_
> > > convenient would be some kind of bot whom I could tell e.g.
> > >
> >
> > Indeed it is inconvenient. I am in the lucky situation that I have
> > another machine locally that can be made reasonably quiet without
> > interfering with my worflow. However, in general
> >
> > > @perf-bot compiler perf
> > >
> > > …or more concretely
> > >
> > > @perf-bot compile nofib/spectral/simple/Main.hs
> > >
> > > …or just
> > >
> > > @nofib-bot run
> > >
> > > … or something like that.
> > >
> > > I've noticed that CI now includes a perf-nofib job. But since it
> > > appears to run on a different machine each time, I'm not sure whether
> > > it's actually useful for comparing performance. Could it be made more
> > > useful by running it consistently on the same dedicated machine?
> > >
> > Indeed, we currently don't have a dedicated machine for timings.
> > However, allocations and executable sizes are still useful.
> >
> > Nevertheless, as noted above I think that we should make more of an
> > effort to measure time. I need to do some shuffling of our runners so we
> > have a quiet bare-metal which can be dedicated to performance
> > measurement. I'll try to get to this in the next day or so.
> >
> > > Another question regarding performing compiler perf measurements
> > > locally is which build flavour to use: So far I have used the "perf"
> > > flavour. A problem here is that a full build seems to take close to an
> > > hour. A rebuild with --freeze1 takes ~15 minutes on my machine. Is
> > > this the right flavour to use?
> > >
> > I think perf is the best option for performance measurement (afterall,
> > we want to know what users would see). However, it is indeed a bit
> > painful.
> >
> > > BTW what's the purpose of the profiled GHC modules built with this
> > > flavour which just seem to additionally prolong compile time? I don't
> > > see a ghc-prof binary or similar in _build/stage1/bin.
> > >
> > Indeed; there is little sense in building profiled modules just for
> > performance measurement. However, I don't believe we currently have a
> > build flavour which provides comparable optimisation but without the
> > profiled way. Perhaps we should add one.
> >
> > > Also, what's the status of gipeda? The most recent commit at
> > > https://perf.haskell.org/ghc/ is from "about a year ago"?
> > >
> > Indeed the machine which was previously providing gipeda builds is sadly
> > no longer around; consequently it's on ice at the moment. I would like
> > to get it going again but recently correctness issues have been taking
> > up more time than I would like to admit.
> >
> > > Sorry for this load of questions and complaints! I do believe though
> > > that if work on compiler performance was a bit better documented and
> > > more convenient, we might see even more progress on that front. :)
> > >
> > Quite alright! Typing out the points above made me realize that there is
> > indeed quite a bit of knowledge that the wiki leaves un-said.
> >
> > Cheers,
> >
> > - Ben
> >
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs