<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
Hi Simon,<code></code><code></code><o:p></o:p>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l0 level1 lfo1">There are
two things going on:<o:p></o:p></li>
</ul>
<ul style="margin-top:0cm" type="disc">
<ol style="margin-top:0cm" type="1" start="1">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l1 level2 lfo3">CI perf
measurements<o:p></o:p></li>
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l1 level2 lfo3">Local
machine perf measurements<o:p></o:p></li>
</ol>
</ul>
<p class="MsoListParagraph">I think that they are somehow
handled differently (why?) but they are all muddled up on the
wiki page.</p>
</div>
</blockquote>
<p>They are handled differently because we do not want to compare
local metrics with CI metrics. The exception is when local metrics
don't exist, then we fall back to CI metrics as a baseline (see <a
href="https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/performance-tests#how-baselines-are-calculated">How
baseline metrics are calculated</a>).<br>
</p>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<p class="MsoListParagraph"><o:p></o:p></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l0 level1 lfo1">My goal is
this:
<o:p></o:p></li>
</ul>
<ul style="margin-top:0cm" type="disc">
<ul style="margin-top:0cm" type="circle">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">Start with
a master commit, say from Dec 2019.<o:p></o:p></li>
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">Implement
some change, on a branch.<o:p></o:p></li>
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">sh
validate –legacy (or something else if you like)<o:p></o:p></li>
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">Look at
perf regressions.</li>
</ul>
</ul>
</div>
</blockquote>
Getting to the *raw data* should be easy:
<ol>
<li>Checkout an the <baseline> commit.</li>
<li>Use `git status` to double check git sees a clean working
tree.</li>
<li>Run the performance tests.</li>
<li>Check out your <target> branch.</li>
<li>Use `git status` to double check git sees a clean working tree
(else commit any changes)</li>
<li>Run the performance tests.</li>
<li>Compare metrics (filtering for `local` metrics and outputting
a chart):</li>
</ol>
<p> <code>python3 testsuite/driver/perf_notes.py --chart
chart.html --test-env local <baseline> <target></code></p>
see `<code>python3 testsuite/driver/perf_notes.py --help`</code> for
more filtering options. This doesn't detect regressions
automatically, it only shows you the raw data. Ideally we'd add an
option to the testrunner to let you specify a baseline commit
manually. I suspect that would be close to what you're looking for.<br>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level1 lfo2">I believe I
have first to utter the incantation<o:p></o:p></li>
</ul>
<p class="MsoNormal" style="margin-left:72.0pt"><span
class="line"><span
style="font-size:10.0pt;font-family:"Courier
New"">$ git fetch
<a class="moz-txt-link-freetext" href="https://gitlab.haskell.org/ghc/ghc-performance-notes.git">https://gitlab.haskell.org/ghc/ghc-performance-notes.git</a>
refs/notes/perf:refs/notes/ci/perf</span></span></p>
</div>
</blockquote>
Yes, this fetches the latest CI metrics into your git notes.<br>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<p class="MsoNormal" style="margin-left:72.0pt"><span
class="line"><span
style="font-size:10.0pt;font-family:"Courier
New""><o:p></o:p></span></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level1 lfo2">But then:<o:p></o:p></li>
<ul style="margin-top:0cm" type="circle">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">How do I
ensure that the baseline perf numbers I get relate to the
master commit I started from, back in Dec 2019? I don’t
want numbers from Jan 2020.</li>
</ul>
</ul>
</div>
</blockquote>
see above.<br>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<ul style="margin-top:0cm" type="disc">
<ul style="margin-top:0cm" type="circle">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">If I
rebase my branch on top of HEAD, say, how do I update the
perf baseline numbers to be for HEAD</li>
</ul>
</ul>
</div>
</blockquote>
The test runner should use HEAD's metrics automatically (see <a
moz-do-not-send="true"
href="https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/performance-tests#how-baselines-are-calculated">How
baseline metrics are calculated</a>), though you will need to
fetch CI metrics or run the perf tests locally on HEAD to get the
relevant metrics.<br>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<ul style="margin-top:0cm" type="disc">
<ul style="margin-top:0cm" type="circle">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level2 lfo2">Generally,
how can I tell the commit to which the baseline numbers
relate?</li>
</ul>
</ul>
</div>
</blockquote>
The test runner will output (per test) which baseline commit is used
e.g. "... from local
<div><span style="color: #000000;"> baseline @ HEAD~2" says the
baseline was a local run from 2 commits ago.<br>
</span></div>
<blockquote type="cite"
cite="mid:VI1PR8303MB0093CE2599BAC9AC46395BC1AD340@VI1PR8303MB0093.EURPRD83.prod.outlook.com">
<div class="WordSection1">
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l2 level1 lfo2">Also, in my
tree I have a series of incremental changes; I want to see
if any of them have perf regressions. How do I do that?</li>
</ul>
</div>
</blockquote>
<p>You can run the perf tests on each commit *in commit order*, and
the previous commit will always be used as the baseline. You can
also then chart the results:</p>
<p> <code>python3 testsuite/driver/perf_notes.py --chart
chart.html --test-env local
<oldestCommit>..<newestCommit></code></p>
<o:p></o:p>
<p>Sorry if this is a bit unoptimal, but I Hope that helps<br>
</p>
<p>- David E</p>
<p><br>
</p>
<pre class="moz-signature" cols="72">
--
David Eichmann, Haskell Consultant
Well-Typed LLP, <a class="moz-txt-link-freetext" href="http://www.well-typed.com">http://www.well-typed.com</a>
Registered in England & Wales, OC335890
118 Wymering Mansions, Wymering Road, London W9 2NF, England </pre>
</body>
</html>