Benchmarking experiences: Cabal test vs compiling nofib/spectral/simple/Main.hs

[Andreas Klebinger]

Hello Devs,

When I started to work on GHC a few years back the Wiki recommended
using nofib/spectral/simple/Main.hs as
a test case for compiler performance changes. I've been using this ever
since.

"Recently" the cabal-test (compiling cabal-the-library) has become sort
of a default benchmark for GHC performance.
I've used the Cabal test as well and it's probably a better test case
than nofib/spectral/simple/Main.hs.
I've started using both usually using spectral/simple to benchmark
intermediate changes and then looking
at the cabal test for the final patch at the end. So far I have rarely
seen a large
difference between using cabal or spectral/simple. Sometimes the
magnitude of the effect was different
between the two, but I've never seen one regress/improve while the other
didn't.

Since the topic came up recently in a discussion I wonder if others use
similar means to quickly bench ghc changes
and what your experiences were in terms of simpler benchmarks being
representative compared to the cabal test.

Cheers,
Andreas