performance testing

[David Eichmann]

-------- Forwarded Message --------
Subject: 	Re: performance testing
Date: 	Wed, 11 Mar 2020 12:08:52 +0000
From: 	David Eichmann <davide at well-typed.com>
To: 	Richard Eisenberg <rae at richarde.dev>




> Ah yes -- very helpful. So the baseline is always (implicitly) the 
> parent commit, if that commit has performance info in the set of 
> notes. So if I have this situation (time flows down)
>
> origin/master: blah
> wip/xyz: big cool change (that slows things down)
> attempt1: comment out some of big cool change
> attempt2: comment out more of big cool change
With you so far.
> and I do a perf test on attempt1, it's quite likely that the perf test 
> will*pass*, because it's comparing to the previous commit.
Yes, assuming you actually ran performance tests on the previous commit 
with a clean working tree (remember we don't record performance metrics 
if the git tree has changes, though you'll see a warning in the test 
output in that case). I'd suggest outputting a graph as described in the 
wiki if you're wondering which commits have recorded performance metrics.
> And then when I do a perf test on attempt2 and see a metric*decrease*, 
> that might just be because I've fixed the perf problem... but I 
> haven't actually made an improvement.

Right. The decrease is compared to attempt1, but this may not be a 
decrease compared to earlier commits. Again I'd suggest graphing the 
data if you want to inspect this yourself.


David E



-- 
David Eichmann, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com

Registered in England & Wales, OC335890
118 Wymering Mansions, Wymering Road, London W9 2NF, England

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200311/637b3b94/attachment.html>