Perf notes

[Simon Peyton Jones]

Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master.
Thanks


Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline.
OK.  But they are guaranteed to be 100.0% identical to the ones discovered by CI, aren't they?   So it's just an implementation detail whether the numbers you save are gotten from one run, or another identical one.

I'm still lost about when I can rely on the perf output of CI and when I can't.  I'm really hoping for a simple answer like:

  *   The CI log tells you the comparison between the preceding commit and this one
No ifs, no buts.  Simple!

Incidentally, would it be possible to output a table (in the log) like we get from nofib-analyse.  It looks like this


            Program           Size    Allocs   Runtime   Elapsed  TotalMem

    --------------------------------------------------------------------------------

              boyer          -0.3%     +5.4%     +0.7%     +1.0%      0.0%

           cichelli          -0.3%     +5.9%     -9.9%     -9.5%      0.0%

          compress2          -0.4%     +9.6%     +7.2%     +6.4%      0.0%

        constraints          -0.3%     +0.2%     -3.0%     -3.4%      0.0%

       cryptarithm2          -0.3%     -3.9%     -2.2%     -2.4%      0.0%

             gamteb          -0.4%     +2.5%     +2.8%     +2.8%      0.0%

               life          -0.3%     -2.2%     -4.7%     -4.9%      0.0%

               lift          -0.3%     -0.3%     -0.8%     -0.5%      0.0%

             linear          -0.3%     -0.1%     -4.1%     -4.5%      0.0%

               mate          -0.2%     +1.4%     -2.2%     -1.9%    -14.3%

             parser          -0.3%     -2.1%     -5.4%     -4.6%      0.0%

             puzzle          -0.3%     +2.1%     -6.6%     -6.3%      0.0%

             simple          -0.4%     +2.8%     -3.4%     -3.3%     -2.2%

            veritas          -0.1%     +0.7%     -0.6%     -1.1%      0.0%

       wheel-sieve2          -0.3%    -19.2%    -24.9%    -24.5%    -42.9%

    --------------------------------------------------------------------------------

                Min          -0.4%    -19.2%    -24.9%    -24.5%    -42.9%

                Max          +0.1%     +9.6%     +7.2%     +6.4%    +33.3%

     Geometric Mean          -0.3%     -0.0%     -3.0%     -2.9%     -0.3%

Instantly comprehensible, one line per benchmark.  I find I spent quite a lot of time search manually in the log and manually building a table (or excerpts thereof) looking like this.

I don't have an opinion about the columns, just wanting a table with one line per benchmark, and a number of columns.

Thanks

Simon


From: David Eichmann <davide at well-typed.com>
Sent: 27 June 2020 20:39
To: Simon Peyton Jones <simonpj at microsoft.com>; ghc-devs at haskell.org
Subject: Re: Perf notes


> I thought that wasn't possible.  Isn't that what CI is *for*?

Yes we run CI on MRs, but once merged into master CI is run again. It's only those metrics from CI on master (post merge) that are ultimately uploaded / used as a baseline.

Re the doubling of bytes-allocated on T9803, that's a good point. Due to the recent change in RSA keys, CI is recently failing to upload metrics (e.g. [1])! I'll fix that then see if I can track down where / if the metric has really regressed in master.



[1] "fatal: Could not read from remote repository."  https://gitlab.haskell.org/ghc/ghc/-/jobs/378487<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fjobs%2F378487&data=02%7C01%7Csimonpj%40microsoft.com%7C7e93b1bdba414d2b98ae08d81ad1b2e6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637288835244421907&sdata=6SpTm36gxWqp0bPp78Kz037d%2BykCs2g%2F0GGZnwXyKMs%3D&reserved=0>

--

David Eichmann, Haskell Consultant

Well-Typed LLP, http://www.well-typed.com<https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-typed.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7C7e93b1bdba414d2b98ae08d81ad1b2e6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637288835244431897&sdata=9%2BIEhlNYjNsc%2FopbyMryXa7dG40d79JrWeQVqtZtx8g%3D&reserved=0>



Registered in England & Wales, OC335890

118 Wymering Mansions, Wymering Road, London W9 2NF, England
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20200629/d5f37718/attachment.html>