<div dir="ltr"><div>Salam Muhaimin,<br></div><div><br>So to answer my own question, the current practice seems to be to just eyeball<br><br>&gt; 
  Geometric Mean          -0.0%     -0.0%     -0.3%     -0.1%     +0.1%<br><br></div><div>and if the numbers are within historical epsilons of 0, that means no change.<br><br></div><div>For a moment, I thought this was some erratically behaving VM.<br>

</div><div><br></div><div>The compile times are across-the-board lower, nice!<br></div></div><div class="gmail_extra"><br clear="all"><div>-- Kim-Ee</div>
<br><br><div class="gmail_quote">On Sat, Sep 28, 2013 at 4:39 AM, Muhaimin Ahsan <span dir="ltr">&lt;<a href="mailto:leroux@fezrev.com" target="_blank">leroux@fezrev.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">

<div style="word-wrap:break-word">Kim-Ee,<div><br></div><div>The updated fib-analyse report from a few hours ago is posted here: <a href="https://gist.github.com/leroux/6725810#file-headvordnub-analysis-L2988" target="_blank">https://gist.github.com/leroux/6725810#file-headvordnub-analysis-L2988</a></div>

<div><div>Comment (<a href="http://ghc.haskell.org/trac/ghc/ticket/8173#comment:9" target="_blank">http://ghc.haskell.org/trac/ghc/ticket/8173#comment:9</a>)</div><div><br></div><div>Sorry for the misunderstanding.</div>
<span class="HOEnZb"><font color="#888888"><div>

<br><span style="text-indent:0px;letter-spacing:normal;font-variant:normal;text-align:-webkit-auto;font-style:normal;display:inline!important;font-weight:normal;float:none;line-height:normal;text-transform:none;font-size:medium;white-space:normal;font-family:&#39;Lucida Grande&#39;;word-spacing:0px">Muhaimin</span>

</div>
<br></font></span><div><div><div class="h5"><div>On Sep 27, 2013, at 3:56 PM, Kim-Ee Yeoh &lt;<a href="mailto:ky3@atamo.com" target="_blank">ky3@atamo.com</a>&gt; wrote:</div><br></div></div><blockquote type="cite"><div>

<div class="h5"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 27, 2013 at 8:14 PM, GHC <span dir="ltr">&lt;<a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a>&gt;</span> wrote:<br>



<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow:hidden">A 5% improvement in compile time is remarkable, if it&#39;s true.  Great!  But<br>




 I&#39;m always worried about the noise in compile times measured in seconds.</div></blockquote></div><br></div><div class="gmail_extra">Does anyone else think the noise in runtimes is alarming considering that the following is the fib-analysis of /binary-identical/ programs? <br>



<br></div><div class="gmail_extra">&gt;              Min          -0.1%     -0.0%    -25.4%    -32.2%     -1.3%<br>
&gt;             Max          +0.1%     +0.0%    +19.0%    +22.2%    +10.0%<br><br></div><div class="gmail_extra">Shouldn&#39;t we find an explanation for this before believing the compile time numbers? What would cause these wide swings on the benchmarking machine?<br>



<br></div><div class="gmail_extra">p.s. For the record: Should do more rigorous statistical testing instead of naive percentages, yes?<br></div><div class="gmail_extra"><br><div>-- Kim-Ee</div>
</div></div></div></div><div class="im">
_______________________________________________<br>ghc-tickets mailing list<br><a href="mailto:ghc-tickets@haskell.org" target="_blank">ghc-tickets@haskell.org</a><br><a href="http://www.haskell.org/mailman/listinfo/ghc-tickets" target="_blank">http://www.haskell.org/mailman/listinfo/ghc-tickets</a><br>

</div></blockquote></div><br></div></div><br>_______________________________________________<br>
ghc-devs mailing list<br>
<a href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a><br>
<a href="http://www.haskell.org/mailman/listinfo/ghc-devs" target="_blank">http://www.haskell.org/mailman/listinfo/ghc-devs</a><br>
<br></blockquote></div><br></div>