On CI

[Sebastian Graf]

Re: Performance drift: I opened
https://gitlab.haskell.org/ghc/ghc/-/issues/17658 a while ago with an idea
of how to measure drift a bit better.
It's basically an automatically checked version of "Ben stares at
performance reports every two weeks and sees that T9872 has regressed by
10% since 9.0"

Maybe we can have Marge check for drift and each individual MR for
incremental perf regressions?

Sebastian

Am Mi., 17. März 2021 um 14:40 Uhr schrieb Richard Eisenberg <
rae at richarde.dev>:

>
>
> On Mar 17, 2021, at 6:18 AM, Moritz Angermann <moritz.angermann at gmail.com>
> wrote:
>
> But what do we expect of patch authors? Right now if five people write
> patches to GHC, and each of them eventually manage to get their MRs green,
> after a long review, they finally see it assigned to marge, and then it
> starts failing? Their patch on its own was fine, but their aggregate with
> other people's code leads to regressions? So we now expect all patch
> authors together to try to figure out what happened? Figuring out why
> something regressed is hard enough, and we only have a very few people who
> are actually capable of debugging this. Thus I believe it would end up with
> Ben, Andreas, Matthiew, Simon, ... or someone else from GHC HQ anyway to
> figure out why it regressed, be it in the Review Stage, or dissecting a
> marge aggregate, or on master.
>
>
> I have previously posted against the idea of allowing Marge to accept
> regressions... but the paragraph above is sadly convincing. Maybe Simon is
> right about opening up the windows to, say, be 100% (which would catch a
> 10x regression) instead of infinite, but I'm now convinced that Marge
> should be very generous in allowing regressions -- provided we also have
> some way of monitoring drift over time.
>
> Separately, I've been concerned for some time about the peculiarity of our
> perf tests. For example, I'd be quite happy to accept a 25% regression on
> T9872c if it yielded a 1% improvement on compiling Cabal. T9872 is very
> very very strange! (Maybe if *all* the T9872 tests regressed, I'd be more
> worried.) I would be very happy to learn that some more general,
> representative tests are included in our examinations.
>
> Richard
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20210317/a339b9e9/attachment.html>