Benchmarking experiences: Cabal test vs compiling nofib/spectral/simple/Main.hs

[Sebastian Graf]

Hi Andreas,

I similarly benchmark compiler performance by compiling Cabal, but only
occasionally. I mostly trust ghc/alloc metrics in CI and check Cabal when I
think there's something afoot and/or want to measure runtime, not only
allocations.

I'm inclined to think that for my purposes (testing the impact of
optimisations) the GHC codebase offers sufficient variety to turn up
fundamental regressions, but maybe it makes sense to build some packages
from head.hackage to detect regressions like
https://gitlab.haskell.org/ghc/ghc/-/issues/19203 earlier. It's all a bit
open-ended and I frankly think I wouldn't get done anything if all my
patches would have to get to the bottom of all regressions and improvements
on the entire head.hackage set. I somewhat trust that users will complain
eventually and file a bug report and that our CI efforts mean that compiler
performance will improve in the mean.

Although it's probably more of a tooling problem: I simply don't know how
to collect the compiler performance metrics for arbitrary cabal packages.
If these metrics would be collected as part of CI, maybe as a nightly or
weekly job, it would be easier to get to the bottom of a regression before
it manifests in a released GHC version. But it all depends on how easy that
would be to set up and how many CI cycles it would burn, and I certainly
don't feel like I'm in a position to answer either question.

Cheers,
Sebastian


Am Mi., 20. Jan. 2021 um 15:28 Uhr schrieb Andreas Klebinger <
klebinger.andreas at gmx.at>:

> Hello Devs,
>
> When I started to work on GHC a few years back the Wiki recommended
> using nofib/spectral/simple/Main.hs as
> a test case for compiler performance changes. I've been using this ever
> since.
>
> "Recently" the cabal-test (compiling cabal-the-library) has become sort
> of a default benchmark for GHC performance.
> I've used the Cabal test as well and it's probably a better test case
> than nofib/spectral/simple/Main.hs.
> I've started using both usually using spectral/simple to benchmark
> intermediate changes and then looking
> at the cabal test for the final patch at the end. So far I have rarely
> seen a large
> difference between using cabal or spectral/simple. Sometimes the
> magnitude of the effect was different
> between the two, but I've never seen one regress/improve while the other
> didn't.
>
> Since the topic came up recently in a discussion I wonder if others use
> similar means to quickly bench ghc changes
> and what your experiences were in terms of simpler benchmarks being
> representative compared to the cabal test.
>
> Cheers,
> Andreas
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20210123/e26831cb/attachment.html>