CI: Choice of base commit for perf comparisons

[Joachim Breitner]

Hi,

the new (or “new”?) handling of perf numbers, where CI just magically
records and compares them, without us having to manually edit the
`all.T` files, is a big improvement, thanks!

However, I found the choice of the base commit to compare against
unhelpful. Assume master is at commit M, and I start a feature branch
and MR with commit A. CI runs, and tells me about a performance
regressions, and CI is red. I now fix the issue and push commit B to
the branch. CI runs, but it picks A to compare against, and now it is
red because of an seemingly unexpected performance improvement!

I would have expected that all CI runs for this MR to compare the
performance against the base branch on master, and to look for perf
change notices in all commit messages in between.

I see these advantages:

 * The reported perf changes correspond to the changes shown on the MR 
   page
 * Green CI = the MR is ready (after squashing)
 * CI will have numbers for the base commit more reliably
   (else, if I push commit C quickly after B, then the job for B might
   be cancelled and Ci will report changes of C against A instead of B,
   which is unexpected).

I have used this logic of reporting perf changes (or any other
“differential CI”) against the base branch in the Motoko project and it
was quite natural.

Would it be desirable and possible for us here, too?


(A possible rebuttal might be: we don’t push new commits to feature
branches, but always squash and rebase, as that’s what we have to do
before merging anyways. If that’s the case then ok, although I
generally lean to having chronological commits on feature branches and
a nice squashed commit on master.)

Cheers,
Joachim


-- 
Joachim Breitner
  mail at joachim-breitner.de
  http://www.joachim-breitner.de/