<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>> I'd be quite happy to accept a 25% regression on T9872c if
it yielded a 1% improvement on compiling Cabal. T9872 is very very
very strange! (Maybe if *all* the T9872 tests regressed, I'd be
more worried.)</p>
<p>While I fully agree with this. We should *always* want to know if
a small syntetic benchmark regresses by a lot.<br>
Or in other words we don't want CI to accept such a regression for
us ever, but the developer of a patch should need to explicitly ok
it.</p>
<p>Otherwise we just slow down a lot of seldom-used code paths by a
lot.<br>
</p>
<p>Now that isn't really an issue anyway I think. The question is
rather is 2% a large enough regression to worry about? 5%? 10%?<br>
</p>
<p>Cheers,<br>
Andreas<br>
</p>
<div class="moz-cite-prefix">Am 17/03/2021 um 14:39 schrieb Richard
Eisenberg:<br>
</div>
<blockquote type="cite"
cite="mid:010f01784069544b-55202ccf-7239-4333-80c6-3f3cd8543527-000000@us-east-2.amazonses.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Mar 17, 2021, at 6:18 AM, Moritz Angermann
<<a href="mailto:moritz.angermann@gmail.com" class=""
moz-do-not-send="true">moritz.angermann@gmail.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="caret-color: rgb(0, 0, 0);
font-family: Helvetica; font-size: 12px; font-style:
normal; font-variant-caps: normal; font-weight: normal;
letter-spacing: normal; text-align: start; text-indent:
0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration: none; float: none; display: inline
!important;" class="">But what do we expect of patch
authors? Right now if five people write patches to GHC,
and each of them eventually manage to get their MRs green,
after a long review, they finally see it assigned to
marge, and then it starts failing? Their patch on its own
was fine, but their aggregate with other people's code
leads to regressions? So we now expect all patch authors
together to try to figure out what happened? Figuring out
why something regressed is hard enough, and we only have a
very few people who are actually capable of debugging
this. Thus I believe it would end up with Ben, Andreas,
Matthiew, Simon, ... or someone else from GHC HQ anyway to
figure out why it regressed, be it in the Review Stage, or
dissecting a marge aggregate, or on master.</span></div>
</blockquote>
</div>
<br class="">
<div class="">I have previously posted against the idea of
allowing Marge to accept regressions... but the paragraph above
is sadly convincing. Maybe Simon is right about opening up the
windows to, say, be 100% (which would catch a 10x regression)
instead of infinite, but I'm now convinced that Marge should be
very generous in allowing regressions -- provided we also have
some way of monitoring drift over time.</div>
<div class=""><br class="">
</div>
<div class="">Separately, I've been concerned for some time about
the peculiarity of our perf tests. For example, I'd be quite
happy to accept a 25% regression on T9872c if it yielded a 1%
improvement on compiling Cabal. T9872 is very very very strange!
(Maybe if *all* the T9872 tests regressed, I'd be more worried.)
I would be very happy to learn that some more general,
representative tests are included in our examinations.</div>
<div class=""><br class="">
</div>
<div class="">Richard</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
ghc-devs mailing list
<a class="moz-txt-link-abbreviated" href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a>
<a class="moz-txt-link-freetext" href="http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a>
</pre>
</blockquote>
</body>
</html>