Nofib runtimes sensitivity to RTS changes

[Joachim Breitner]

Hi,

this is mostly a rant, targeted at the very vague fact that runtime
performance benchmarks are so damn hard.

This commit just came in:
https://git.haskell.org/ghc.git/commitdiff/56c9bb39246f9ffd8ed41a0656bfe8e60d23be57
Clearly, only parts of the profiling RTS have changed.

This is what perf.haskell.org measures:
https://perf.haskell.org/ghc/#revision/56c9bb39246f9ffd8ed41a0656bfe8e60d23be57
Highlights:
    nofib/time/cryptarithm1     	    0.502     	    + 9.36%     	    0.549     	    seconds
    nofib/time/fasta             	    0.406     	    + 4.43%     	    0.424     	    seconds
These changes are permanent, i.e. the following commits consistently
give the higher runtimes.

But nofib does not even run the profiling RTS!

*very audible sigh*

(Conclusion: Ignore runtime changes in nofib when the RTS has changed,
the results are simply unusable.)

I also wonder if we can adjust the input and parameters to nofib (in a
new “veryslow” setting) to get them all run in a few seconds on modern
hardware, instead of subseconds.

Greetings,
Joachim


-- 
Joachim “nomeata” Breitner
  mail at joachim-breitner.de • https://www.joachim-breitner.de/
  XMPP: nomeata at joachim-breitner.de • OpenPGP-Key: 0xF0FBF51F
  Debian Developer: nomeata at debian.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: This is a digitally signed message part
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20170212/60c2c22c/attachment.sig>