Changes to performance testing?

[John Ericson]

If we make the changes proposed in the "On CI" thread, I think we solve 
most of this for free. If a perf test fails, resubmitting should not 
need to rebuild GHC because it is cached. I'd say at that point, it's 
not even worth making Marge bot auto-accept extra improvements because 
restarting with new perf windows should be so easy.

John

On 2/22/21 8:46 AM, Andreas Klebinger wrote:
>
> This seems quite reasonable to me.
> Not sure about the cost of implementing it (and the feasability of it 
> if/when merge-trains arrive).
>
> Andreas
>
> Am 21/02/2021 um 21:31 schrieb Richard Eisenberg:
>>
>>
>>> On Feb 21, 2021, at 11:24 AM, Ben Gamari <ben at well-typed.com 
>>> <mailto:ben at well-typed.com>> wrote:
>>>
>>> To mitigate this I would suggest that we allow performance test failures
>>> in marge-bot pipelines. A slightly weaker variant of this idea would
>>> instead only allow performance *improvements*. I suspect the latter
>>> would get most of the benefit, while eliminating the possibility that a
>>> large regression goes unnoticed.
>>
>> The value in making performance improvements a test failure is so 
>> that patch authors can be informed of what they have done, to make 
>> sure it matches expectations. This need can reasonably be satisfied 
>> without stopping merging. That is, if Marge can accept performance 
>> improvements, while (say) posting to each MR involved that it may 
>> have contributed to a performance improvement, then I think we've 
>> done our job here.
>>
>> On the other hand, a performance degradation is a bug, just like, 
>> say, an error message regression. Even if it's a combination of 
>> commits that cause the problem (an actual possibility even for error 
>> message regressions), it's still a bug that we need to either fix or 
>> accept (balanced out by other improvements). The pain of debugging 
>> this scenario might be mitigated if there were a collation of the 
>> performance wibbles for each individual commit. This information is, 
>> in general, available: each commit passed CI on its own, and so it 
>> should be possible to create a little report with its rows being perf 
>> tests and its columns being commits or MR #s; each cell in the table 
>> would be a percentage regression. If we're lucky, the regression 
>> Marge sees will be the sum(*) of the entries in one of the rows -- 
>> this means that we have a simple agglomeration of performance 
>> degradation. If we're less lucky, the whole will not equal the sum of 
>> the parts, and some of the patches interfere. In either case, the 
>> table would suggest a likely place to look next.
>>
>> (*) I suppose if we're recording percentages, it wouldn't necessarily 
>> be the actual sum, because percentages are a bit funny. But you get 
>> my meaning.
>>
>> Pulling this all together:
>> * I'm against the initial proposal of allowing all performance 
>> failures by Marge. This will allow bugs to accumulate (in my opinion).
>> * I'm in favor of allowing performance improvements to be accepted by 
>> Marge.
>> * To mitigate against the information loss of Marge accepting 
>> performance improvements, it would be great if Marge could alert MR 
>> authors that a cumulative performance improvement took place.
>> * To mitigate against the annoyance of finding a performance 
>> regression in a merge commit that does not appear in any component 
>> commit, it would be great if there were a tool to collect performance 
>> numbers from a set of commits and present them in a table for further 
>> analysis.
>>
>> These "mitigations" might take work. If labor is impossible to 
>> produce to complete this work, I'm in favor of simply allowing the 
>> performance improvements, maybe also filing a ticket about these 
>> potential improvements to the process.
>>
>> Richard
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20210223/6a280621/attachment.html>