klebinger.andreas at gmx.at
Wed Mar 24 13:50:09 UTC 2021
> What about the case where the rebase *lessens* the improvement? That
is, you're expecting these 10 cases to improve, but after a rebase, only
1 improves. That's news! But a blanket "accept improvements" won't tell you.
I don't think that scenario currently triggers a CI failure. So this
wouldn't really change.
As I understand it the current logic is:
* Run tests
* Check if any cross the metric thresholds set in the test.
* If so check if that test is allowed to cross the threshold.
I believe we don't check that all benchmarks listed with an expected
in/decrease actually do so.
It would also be hard to do so reasonably without making it even harder
to push MRs through CI.
Am 24/03/2021 um 13:08 schrieb Richard Eisenberg:
> What about the case where the rebase *lessens* the improvement? That is, you're expecting these 10 cases to improve, but after a rebase, only 1 improves. That's news! But a blanket "accept improvements" won't tell you.
> I'm not hard against this proposal, because I know precise tracking has its own costs. Just wanted to bring up another scenario that might be factored in.
>> On Mar 24, 2021, at 7:44 AM, Andreas Klebinger <klebinger.andreas at gmx.at> wrote:
>> After the idea of letting marge accept unexpected perf improvements and
>> looking at https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4759
>> which failed because of a single test, for a single build flavour
>> crossing the
>> improvement threshold where CI fails after rebasing I wondered.
>> When would accepting a unexpected perf improvement ever backfire?
>> In practice I either have a patch that I expect to improve performance
>> for some things
>> so I want to accept whatever gains I get. Or I don't expect improvements
>> so it's *maybe*
>> worth failing CI for in case I optimized away some code I shouldn't or
>> something of that
>> How could this be actionable? Perhaps having a set of indicator for CI of
>> "Accept allocation decreases"
>> "Accept residency decreases"
>> Would be saner. I have personally *never* gotten value out of the
>> to list the indivial tests that improve. Usually a whole lot of them do.
>> Some cross
>> the threshold so I add them. If I'm unlucky I have to rebase and a new
>> one might
>> make it across the threshold.
>> Being able to accept improvements (but not regressions) wholesale might be a
>> reasonable alternative.
>> ghc-devs mailing list
>> ghc-devs at haskell.org
More information about the ghc-devs