CI: Choice of base commit for perf comparisons
Joachim Breitner
mail at joachim-breitner.de
Wed Dec 22 14:07:17 UTC 2021
Thanks! I like it when my feature suggestions are implemented even before I voice them ;-)
22.12.2021 14:13:24 Richard Eisenberg <lists at richarde.dev>:
> It seems to be that this thought is in the air right now. This was done just a few days ago: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7184
>
> https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7231 also looks relevant.
>
> Richard
>
>> On Dec 22, 2021, at 7:19 AM, Joachim Breitner <mail at joachim-breitner.de> wrote:
>>
>> Hi,
>>
>> the new (or “new”?) handling of perf numbers, where CI just magically
>> records and compares them, without us having to manually edit the
>> `all.T` files, is a big improvement, thanks!
>>
>> However, I found the choice of the base commit to compare against
>> unhelpful. Assume master is at commit M, and I start a feature branch
>> and MR with commit A. CI runs, and tells me about a performance
>> regressions, and CI is red. I now fix the issue and push commit B to
>> the branch. CI runs, but it picks A to compare against, and now it is
>> red because of an seemingly unexpected performance improvement!
>>
>> I would have expected that all CI runs for this MR to compare the
>> performance against the base branch on master, and to look for perf
>> change notices in all commit messages in between.
>>
>> I see these advantages:
>>
>> * The reported perf changes correspond to the changes shown on the MR
>> page
>> * Green CI = the MR is ready (after squashing)
>> * CI will have numbers for the base commit more reliably
>> (else, if I push commit C quickly after B, then the job for B might
>> be cancelled and Ci will report changes of C against A instead of B,
>> which is unexpected).
>>
>> I have used this logic of reporting perf changes (or any other
>> “differential CI”) against the base branch in the Motoko project and it
>> was quite natural.
>>
>> Would it be desirable and possible for us here, too?
>>
>>
>> (A possible rebuttal might be: we don’t push new commits to feature
>> branches, but always squash and rebase, as that’s what we have to do
>> before merging anyways. If that’s the case then ok, although I
>> generally lean to having chronological commits on feature branches and
>> a nice squashed commit on master.)
>>
>> Cheers,
>> Joachim
>>
>>
>> --
>> Joachim Breitner
>> mail at joachim-breitner.de
>> http://www.joachim-breitner.de/
>>
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
More information about the ghc-devs
mailing list