performance testing

[Richard Eisenberg]

Ah yes -- very helpful. So the baseline is always (implicitly) the parent commit, if that commit has performance info in the set of notes. So if I have this situation (time flows down)

origin/master: blah
wip/xyz: big cool change (that slows things down)
attempt1: comment out some of big cool change
attempt2: comment out more of big cool change

and I do a perf test on attempt1, it's quite likely that the perf test will *pass*, because it's comparing to the previous commit. And then when I do a perf test on attempt2 and see a metric *decrease*, that might just be because I've fixed the perf problem... but I haven't actually made an improvement.

(This story is true. Names have been changed to protect the innocent.)

Is my understanding accurate here?

Sorry for not finding the appropriate wiki page sooner. It felt like I was witnessing non-determinism, but it actually makes sense now.

Thanks!
Richard

> On Mar 11, 2020, at 9:59 AM, David Eichmann <davide at well-typed.com> wrote:
> 
> Richard
> 
> The performance tests seem to be causing more confusion than I'd of liked. A baseline is established from previous performance test runs. There is a wiki page that may help: https://gitlab.haskell.org/ghc/ghc/wikis/building/running-tests/performance-tests. Let me know if that's not sufficiently helpful.
> 
> David E
> 
> On 3/10/20 11:05 PM, Richard Eisenberg wrote:
>> Hi all,
>> 
>> I'm very confused by how to do performance testing.
>> 
>> I have a patch that seems to cause a performance regression. So I built the patch locally (`validate` flavor, with `make`) and then reproduced the regression. Good. Then, I make a small change, rebuild, commit, and then test. But now it seems that the baseline has changed. (This took some time to discover after seemingly non-deterministic results!)
>> 
>> So: how can I make several different commits, with different experiments, switch between them at will, all without changing my baseline? I do *not* have the magic "metric decrease" bit in my commit message. Or, at least, I didn't put it there.
>> 
>> Thanks!
>> Richard
>> _______________________________________________
>> ghc-devs mailing list
>> ghc-devs at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> 
> -- 
> David Eichmann, Haskell Consultant
> Well-Typed LLP, http://www.well-typed.com
> 
> Registered in England & Wales, OC335890
> 118 Wymering Mansions, Wymering Road, London W9 2NF, England
> 
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs