<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>Hello,</p>
<p>So CI metrics are being pushed again, good. The immediate issue
was perf test <span lang="EN-US">T9803. I've looked at metrics
across all CI test environments and the last 300 commit on
master. Metric looks stable. I've attached the output of:
`python3 ./testsuite/driver/perf_notes.py --chart T9203.html
--ci --test-name T9203 origin/master~300..origin/master`.</span></p>
<p><span lang="EN-US">Simon, you are on windows, correct? If so the
result you posted is using the "</span><span lang="EN-US"><span
lang="EN-US">x86_64-linux-deb9" environment as a baseline when
it should be using "x86_64" which is much closer to the result
you're getting.</span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US">>
</span></span><span lang="EN-US"><span
style="mso-fareast-language:EN-US"><span lang="EN-US"><span
style="mso-fareast-language:EN-US">So it’s just an
implementation detail whether the numbers you save are
gotten from one run, or another identical one.</span></span>
<br>
</span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US">For
the most part yes, but since we usually rebase and batch
commit with marge bot, we are really creating a new commit. So
it makes some sense to rerun CI.</span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US">>
</span></span><span lang="EN-US"><span
style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">The CI log tells you the
comparison between the preceding commit and this one</span></span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">I'd love for this to be
the behavior too. The problem is, most of the time we don't
have the metrics for the parent commit, and generating them
is expensive. We could automatically checkout / build the
previous commit and run perf tests, but this doesn't seem
like good design. That's why we've resorted to searching for
"most recent" local or CI metrics to establish an
approximate baseline. Requiring the developers to always run
perf tests locally on the previous commit will be extremely
annoying. Another option is to just disable perf tests by
default.</span></span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">It's not satisfying but
it's hard to think of a better solution. I think the current
implementation is still more convenient than before, where
baselines were just hard coded into test files. If you
wanted to investigate a particular commit, you had to
manually establish a baseline. Perhaps the problem with this
new system is that it's a bit too "magical" and it's unclear
how to interpret results. Perhaps this can be remedied with
better output form the test runner.</span></span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">> W</span></span></span><span
lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">ould it be possible to
output a table (in the log) like we get from nofib-analyse</span></span></span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">Absolutely. This should
be fairly easy to implement. I've created #18417.</span></span></span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US">- David E<br>
</span></span></span></span></p>
<p><span lang="EN-US"><span style="mso-fareast-language:EN-US"><span
style="mso-fareast-language:EN-US"><br>
</span></span></span></p>
<div class="moz-cite-prefix">On 6/29/20 3:08 PM, Simon Peyton Jones
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:VI1PR83MB04293F9A22AA4372DFF0B76CAD6E0@VI1PR83MB0429.EURPRD83.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.Code, li.Code, div.Code
{mso-style-name:Code;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:22.7pt;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
font-weight:bold;}
span.ws-pre-wrap
{mso-style-name:ws-pre-wrap;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1200700991;
mso-list-type:hybrid;
mso-list-template-ids:-2060308090 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p style="margin-left:36.0pt"><span lang="EN-US">Re the doubling
of bytes-allocated on T9803, that's a good point. Due to the
recent change in RSA keys, CI is recently failing to upload
metrics (e.g. [1])! I'll fix that then see if I can track
down where / if the metric has really regressed in master.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span lang="EN-US">Yes we run CI
on MRs, but once merged into master CI is run again. It's
only those metrics from CI on master (post merge) that are
ultimately uploaded / used as a baseline.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">OK.
But they are guaranteed to be 100.0% identical to the ones
discovered by CI, aren’t they? So it’s just an
implementation detail whether the numbers you save are
gotten from one run, or another identical one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I’m
still lost about when I can rely on the perf output of CI
and when I can’t. I’m really hoping for a simple answer
like:<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph"
style="margin-left:0cm;mso-list:l0 level1 lfo1"><span
style="mso-fareast-language:EN-US">The CI log tells you
the comparison between the preceding commit and this one<o:p></o:p></span></li>
</ul>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">No
ifs, no buts. Simple!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Incidentally,
would it be possible to output a table (in the log) like we
get from nofib-analyse. It looks like this<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
Program Size Allocs Runtime Elapsed
TotalMem<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
--------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
boyer -0.3% +5.4% +0.7% +1.0% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
cichelli -0.3% +5.9% -9.9% -9.5%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
compress2 -0.4% +9.6% +7.2% +6.4%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
constraints -0.3% +0.2% -3.0%
-3.4% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
cryptarithm2 -0.3% -3.9% -2.2%
-2.4% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
gamteb -0.4% +2.5% +2.8% +2.8%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
life -0.3% -2.2% -4.7% -4.9% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
lift -0.3% -0.3% -0.8% -0.5% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
linear -0.3% -0.1% -4.1% -4.5%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
mate -0.2% +1.4% -2.2% -1.9% -14.3%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
parser -0.3% -2.1% -5.4% -4.6%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
puzzle -0.3% +2.1% -6.6% -6.3%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
simple -0.4% +2.8% -3.4% -3.3%
-2.2%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
veritas -0.1% +0.7% -0.6% -1.1%
0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
wheel-sieve2 -0.3% -19.2% -24.9% -24.5%
-42.9%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
--------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
Min -0.4% -19.2% -24.9% -24.5% -42.9%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
Max +0.1% +9.6% +7.2% +6.4% +33.3%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US">
Geometric Mean -0.3% -0.0% -3.0%
-2.9% -0.3%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Instantly
comprehensible, one line per benchmark. I find I spent
quite a lot of time search manually in the log and manually
building a table (or excerpts thereof) looking like this.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I
don’t have an opinion about the columns, just wanting a
table with one line per benchmark, and a number of columns.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Simon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm
0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span
lang="EN-US"> David Eichmann
<a class="moz-txt-link-rfc2396E" href="mailto:davide@well-typed.com"><davide@well-typed.com></a>
<br>
<b>Sent:</b> 27 June 2020 20:39<br>
<b>To:</b> Simon Peyton Jones
<a class="moz-txt-link-rfc2396E" href="mailto:simonpj@microsoft.com"><simonpj@microsoft.com></a>; <a class="moz-txt-link-abbreviated" href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a><br>
<b>Subject:</b> Re: Perf notes<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span lang="EN-US">> I thought that wasn’t possible.
Isn’t that what CI is *<b>for</b>*?</span><o:p></o:p></p>
<p><span lang="EN-US">Yes we run CI on MRs, but once merged
into master CI is run again. It's only those metrics from
CI on master (post merge) that are ultimately uploaded /
used as a baseline.</span><o:p></o:p></p>
<p><span lang="EN-US">Re the doubling of bytes-allocated on
T9803, that's a good point. Due to the recent change in
RSA keys, CI is recently failing to upload metrics (e.g.
[1])! I'll fix that then see if I can track down where /
if the metric has really regressed in master.</span><o:p></o:p></p>
<p><o:p> </o:p></p>
<p><span lang="EN-US">[1] "</span><span class="ws-pre-wrap"><span
style="font-size:10.0pt;font-family:"Courier
New"" lang="EN-US">fatal: Could not read from
remote repository." </span></span><span lang="EN-US">
<a
href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fjobs%2F378487&data=02%7C01%7Csimonpj%40microsoft.com%7C7e93b1bdba414d2b98ae08d81ad1b2e6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637288835244421907&sdata=6SpTm36gxWqp0bPp78Kz037d%2BykCs2g%2F0GGZnwXyKMs%3D&reserved=0"
moz-do-not-send="true">
https://gitlab.haskell.org/ghc/ghc/-/jobs/378487</a></span><o:p></o:p></p>
<p>-- <o:p></o:p></p>
<pre>David Eichmann, Haskell Consultant<o:p></o:p></pre>
<pre>Well-Typed LLP, <a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-typed.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C7e93b1bdba414d2b98ae08d81ad1b2e6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637288835244431897&sdata=9%2BIEhlNYjNsc%2FopbyMryXa7dG40d79JrWeQVqtZtx8g%3D&reserved=0" moz-do-not-send="true">http://www.well-typed.com</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Registered in England & Wales, OC335890<o:p></o:p></pre>
<pre>118 Wymering Mansions, Wymering Road, London W9 2NF, England <o:p></o:p></pre>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
David Eichmann, Haskell Consultant
Well-Typed LLP, <a class="moz-txt-link-freetext" href="http://www.well-typed.com">http://www.well-typed.com</a>
Registered in England & Wales, OC335890
118 Wymering Mansions, Wymering Road, London W9 2NF, England </pre>
</body>
</html>