CI: Choice of base commit for perf comparisons

[Richard Eisenberg]

It seems to be that this thought is in the air right now. This was done just a few days ago: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7184

https://gitlab.haskell.org/ghc/ghc/-/merge_requests/7231 also looks relevant.

Richard

> On Dec 22, 2021, at 7:19 AM, Joachim Breitner <mail at joachim-breitner.de> wrote:
> 
> Hi,
> 
> the new (or “new”?) handling of perf numbers, where CI just magically
> records and compares them, without us having to manually edit the
> `all.T` files, is a big improvement, thanks!
> 
> However, I found the choice of the base commit to compare against
> unhelpful. Assume master is at commit M, and I start a feature branch
> and MR with commit A. CI runs, and tells me about a performance
> regressions, and CI is red. I now fix the issue and push commit B to
> the branch. CI runs, but it picks A to compare against, and now it is
> red because of an seemingly unexpected performance improvement!
> 
> I would have expected that all CI runs for this MR to compare the
> performance against the base branch on master, and to look for perf
> change notices in all commit messages in between.
> 
> I see these advantages:
> 
> * The reported perf changes correspond to the changes shown on the MR 
>   page
> * Green CI = the MR is ready (after squashing)
> * CI will have numbers for the base commit more reliably
>   (else, if I push commit C quickly after B, then the job for B might
>   be cancelled and Ci will report changes of C against A instead of B,
>   which is unexpected).
> 
> I have used this logic of reporting perf changes (or any other
> “differential CI”) against the base branch in the Motoko project and it
> was quite natural.
> 
> Would it be desirable and possible for us here, too?
> 
> 
> (A possible rebuttal might be: we don’t push new commits to feature
> branches, but always squash and rebase, as that’s what we have to do
> before merging anyways. If that’s the case then ok, although I
> generally lean to having chronological commits on feature branches and
> a nice squashed commit on master.)
> 
> Cheers,
> Joachim
> 
> 
> -- 
> Joachim Breitner
>  mail at joachim-breitner.de
>  http://www.joachim-breitner.de/
> 
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs