<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.Code, li.Code, div.Code
{mso-style-name:Code;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:22.7pt;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
font-weight:bold;}
span.ws-pre-wrap
{mso-style-name:ws-pre-wrap;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1200700991;
mso-list-type:hybrid;
mso-list-template-ids:-2060308090 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p style="margin-left:36.0pt"><span lang="EN-US">Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where /
if the metric has really regressed in master.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p style="margin-left:36.0pt"><span lang="EN-US">Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">OK. But they are guaranteed to be 100.0% identical to the ones discovered by CI, aren’t they? So it’s just an implementation detail whether the numbers you save are gotten from one run, or another
identical one.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I’m still lost about when I can rely on the perf output of CI and when I can’t. I’m really hoping for a simple answer like:<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1"><span style="mso-fareast-language:EN-US">The CI log tells you the comparison between the preceding commit and this one<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">No ifs, no buts. Simple!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Incidentally, would it be possible to output a table (in the log) like we get from nofib-analyse. It looks like this<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> Program Size Allocs Runtime Elapsed TotalMem<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> --------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> boyer -0.3% +5.4% +0.7% +1.0% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> cichelli -0.3% +5.9% -9.9% -9.5% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> compress2 -0.4% +9.6% +7.2% +6.4% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> constraints -0.3% +0.2% -3.0% -3.4% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> cryptarithm2 -0.3% -3.9% -2.2% -2.4% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> gamteb -0.4% +2.5% +2.8% +2.8% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> life -0.3% -2.2% -4.7% -4.9% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> lift -0.3% -0.3% -0.8% -0.5% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> linear -0.3% -0.1% -4.1% -4.5% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> mate -0.2% +1.4% -2.2% -1.9% -14.3%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> parser -0.3% -2.1% -5.4% -4.6% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> puzzle -0.3% +2.1% -6.6% -6.3% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> simple -0.4% +2.8% -3.4% -3.3% -2.2%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> veritas -0.1% +0.7% -0.6% -1.1% 0.0%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> wheel-sieve2 -0.3% -19.2% -24.9% -24.5% -42.9%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> --------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> Min -0.4% -19.2% -24.9% -24.5% -42.9%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> Max +0.1% +9.6% +7.2% +6.4% +33.3%<o:p></o:p></span></p>
<p class="Code"><span style="mso-fareast-language:EN-US"> Geometric Mean -0.3% -0.0% -3.0% -2.9% -0.3%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Instantly comprehensible, one line per benchmark. I find I spent quite a lot of time search manually in the log and manually building a table (or excerpts thereof) looking like this.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">I don’t have an opinion about the columns, just wanting a table with one line per benchmark, and a number of columns.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Thanks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Simon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> David Eichmann <davide@well-typed.com>
<br>
<b>Sent:</b> 27 June 2020 20:39<br>
<b>To:</b> Simon Peyton Jones <simonpj@microsoft.com>; ghc-devs@haskell.org<br>
<b>Subject:</b> Re: Perf notes<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span lang="EN-US">> I thought that wasn’t possible. Isn’t that what CI is *<b>for</b>*?</span><o:p></o:p></p>
<p><span lang="EN-US">Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline.</span><o:p></o:p></p>
<p><span lang="EN-US">Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed
in master.</span><o:p></o:p></p>
<p><o:p> </o:p></p>
<p><span lang="EN-US">[1] "</span><span class="ws-pre-wrap"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">fatal: Could not read from remote repository." </span></span><span lang="EN-US">
<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fjobs%2F378487&data=02%7C01%7Csimonpj%40microsoft.com%7C7e93b1bdba414d2b98ae08d81ad1b2e6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637288835244421907&sdata=6SpTm36gxWqp0bPp78Kz037d%2BykCs2g%2F0GGZnwXyKMs%3D&reserved=0">
https://gitlab.haskell.org/ghc/ghc/-/jobs/378487</a></span><o:p></o:p></p>
<p>-- <o:p></o:p></p>
<pre>David Eichmann, Haskell Consultant<o:p></o:p></pre>
<pre>Well-Typed LLP, <a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-typed.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C7e93b1bdba414d2b98ae08d81ad1b2e6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637288835244431897&sdata=9%2BIEhlNYjNsc%2FopbyMryXa7dG40d79JrWeQVqtZtx8g%3D&reserved=0">http://www.well-typed.com</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Registered in England & Wales, OC335890<o:p></o:p></pre>
<pre>118 Wymering Mansions, Wymering Road, London W9 2NF, England <o:p></o:p></pre>
</div>
</div>
</body>
</html>