Measuring performance of GHC

[Moritz Angermann]

> |  - One of the core issues I see in day to day programming (even though
> |    not necessarily with haskell right now) is that the spare time I
> |  have
> |    to file bug reports, boil down performance regressions etc. and file
> |    them with open source projects is not paid for and hence minimal.
> |    Hence whenever the tools I use make it really easy for me to file a
> |    bug, performance regression or fix something that takes the least
> |  time
> |    the chances of me being able to help out increase greatly.  This was
> |  one
> |    of the ideas behind using just pull requests.
> |    E.g. This code seems to be really slow, or has subjectively
> |  regressed in
> |    compilation time. I also feel confident I can legally share this
> |  code
> |    snipped. So I just create a quick pull request with a short
> |  description,
> |    and then carry on with what ever pressing task I’m trying to solve
> |  right
> |    now.
> 
> There's the same difficulty at the other end too - people who might fix perf regressions are typically not paid for either.  So they (eg me) tend to focus on things where there is a small repro case, which in turn costs work to produce.  Eg #12745 which I fixed recently in part because thomie found a lovely small example.
> 
> So I'm a bit concerned that lowering the barrier to entry for perf reports might not actually lead to better perf.  (But undeniably the suite we built up would be a Good Thing, so we'd be a bit further forward.)
> 
> Simon

I did not intend to imply that there was a surplus of time on the other end :)

If this would result in a bunch of tiny test cases that can pinpoint the
underlying issue, I’m not certain.  Say we would tag the test cases though
(e.g. uses TH, uses GADTs, uses X, Y and Z) and run these samples on every
commit or every other commit (what ever the available hardware would allow the
test suite to run on (and maybe even backtest where possible)) regressions w.r.t.
subsets might be identifiable. E.g. commit <hash> made testcases predominantly
with GADTs spike.

Worst case scenario we have to declare defeat and decide that this approach 
has not produced any viable results, and we wasted time of contributes providing
the samples.  On the other hand we would never know without the samples, as they
would have never been provided in the first place?

Cheers,
 moritz