Nofib runtimes sensitivity to RTS changes

[Ben Gamari]

Joachim Breitner <mail at joachim-breitner.de> writes:

> Hi,
>
> this is mostly a rant, targeted at the very vague fact that runtime
> performance benchmarks are so damn hard.
>
> This commit just came in:
> https://git.haskell.org/ghc.git/commitdiff/56c9bb39246f9ffd8ed41a0656bfe8e60d23be57
> Clearly, only parts of the profiling RTS have changed.
>
> This is what perf.haskell.org measures:
> https://perf.haskell.org/ghc/#revision/56c9bb39246f9ffd8ed41a0656bfe8e60d23be57
> Highlights:
>     nofib/time/cryptarithm1     	    0.502     	    + 9.36%     	    0.549     	    seconds
>     nofib/time/fasta             	    0.406     	    + 4.43%     	    0.424     	    seconds
> These changes are permanent, i.e. the following commits consistently
> give the higher runtimes.
>
> But nofib does not even run the profiling RTS!
>
> *very audible sigh*
>
> (Conclusion: Ignore runtime changes in nofib when the RTS has changed,
> the results are simply unusable.)
>
> I also wonder if we can adjust the input and parameters to nofib (in a
> new “veryslow” setting) to get them all run in a few seconds on modern
> hardware, instead of subseconds.

I think this would be a very worthwhile effort. Sub-second runtimes
really are just plain useless.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20170212/f01b2519/attachment.sig>