[GHC] #15357: Make nofib suitable for runtime measurements.
GHC
ghc-devs at haskell.org
Mon Jul 9 16:31:58 UTC 2018
#15357: Make nofib suitable for runtime measurements.
-------------------------------------+-------------------------------------
Reporter: AndreasK | Owner: (none)
Type: task | Status: new
Priority: normal | Milestone: 8.6.1
Component: NoFib benchmark | Version: 8.4.3
suite |
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: | Blocking:
Related Tickets: | Differential Rev(s):
Wiki Page: |
-------------------------------------+-------------------------------------
Old description:
> Currently many benchmarks in a default nofib run have runtimes so low
> that they become meaningless.
>
> This comes with a bunch of issues:
> * It takes up build/runtime while not actually improving the accuracy of
> runtime measurements.
> * It skews results as the jump from 1ms to 2ms registers as a 200%
> change.
> * It can hide regression which fall outside of the measured granularity.
> * Higher number of runs per benchmark spend far too much time on the few
> slower benchmarks.
>
> At the extreme end there are 9 benchmarks with runtimes shorter than the
> granularity of 1ms.
>
> For some benchmarks changing this might be as easy as increasing the
> number of runs/iterations.
> For others changing problem sizes is not as easy since they process more
> complex input data.
>
> Below are the runtimes of a recent default nofib run.
>
> ||= benchmark =|| Time (ms) ||
> || ansi || 0 ||
> || awards || 0 ||
> || banner || 0 ||
> || calendar || 0 ||
> || expert || 0 ||
> || gen_regexps || 0 ||
> || grep || 0 ||
> || pretty || 0 ||
> || scc || 0 ||
> || cse || 1 ||
> || eliza || 1 ||
> || lift || 1 ||
> || mkhprog || 1 ||
> || prolog || 1 ||
> || sorting || 1 ||
> || veritas || 1 ||
> || knights || 2 ||
> || minimax || 2 ||
> || x2n1 || 2 ||
> || mandel2 || 3 ||
> || parstof || 3 ||
> || fluid || 4 ||
> || pic || 4 ||
> || boyer2 || 5 ||
> || bspt || 5 ||
> || cryptarithm2 || 5 ||
> || fish || 6 ||
> || gg || 6 ||
> || reptile || 6 ||
> || symalg || 6 ||
> || tak || 6 ||
> || VSD || 7 ||
> || rfib || 7 ||
> || queens || 9 ||
> || rewrite || 9 ||
> || sched || 11 ||
> || fem || 12 ||
> || fibheaps || 13 ||
> || parser || 14 ||
> || rsa || 14 ||
> || genfft || 15 ||
> || clausify || ansi || 0 ||
> || awards || 0 ||
> || banner || 0 ||
> || calendar || 0 ||
> || expert || 0 ||
> || gen_regexps || 0 ||
> || grep || 0 ||
> || pretty || 0 ||
> || scc || 0 ||
> || cse || 1 ||
> || eliza || 1 ||
> || lift || 1 ||
> || mkhprog || 1 ||
> || prolog || 1 ||
> || sorting || 1 ||
> || veritas || 1 ||
> || knights || 2 ||
> || minimax || 2 ||
> || x2n1 || 2 ||
> || mandel2 || 3 ||
> || parstof || 3 ||
> || fluid || 4 ||
> || pic || 4 ||
> || boyer2 || 5 ||
> || bspt || 5 ||
> || cryptarithm2 || 5 ||
> || fish || 6 ||
> || gg || 6 ||
> || reptile || 6 ||
> || symalg || 6 ||
> || tak || 6 ||
> || VSD || 7 ||
> || rfib || 7 ||
> || queens || 9 ||
> || rewrite || 9 ||
> || sched || 11 ||
> || fem || 12 ||
> || fibheaps || 13 ||
> || parser || 14 ||
> || rsa || 14 ||
> || genfft || 15 ||
> || clausify || 17 ||
> || gamteb || 17 ||
> || fft || 18 ||
> || boyer || 20 ||
> || gcd || 20 ||
> || sphere || 21 ||
> || fft2 || 23 ||
> || infer || 25 ||
> || maillist || 25 ||
> || mandel || 27 ||
> || nucleic2 || 30 ||
> || primes || 34 ||
> || cichelli || 35 ||
> || ida || 39 ||
> || listcompr || 41 ||
> || listcopy || 43 ||
> || multiplier || 43 ||
> || wang || 43 ||
> || hpg || 45 ||
> || anna || 46 ||
> || reverse-complem || 46 ||
> || integrate || 48 ||
> || primetest || 50 ||
> || paraffins || 54 ||
> || solid || 54 ||
> || puzzle || 55 ||
> || treejoin || 61 ||
> || compress2 || 64 ||
> || compress || 65 ||
> || event || 65 ||
> || bernouilli || 74 ||
> || comp_lab_zift || 78 ||
> || simple || 86 ||
> || wheel-sieve2 || 91 ||
> || life || 100 ||
> || exp3_8 || 102 ||
> || wave4main || 103 ||
> || typecheck || 106 ||
> || atom || 110 ||
> || fulsom || 112 ||
> || CS || 116 ||
> || para || 120 ||
> || kahan || 138 ||
> || transform || 145 ||
> || power || 150 ||
> || cacheprof || 151 ||
> || hidden || 160 ||
> || scs || 168 ||
> || lcss || 175 ||
> || wheel-sieve1 || 199 ||
> || VS || 204 ||
> || pidigits || 216 ||
> || FS || 222 ||
> || last-piece || 230 ||
> || fasta || 263 ||
> || cryptarithm1 || 264 ||
> || CSD || 272 ||
> || digits-of-e1 || 273 ||
> || binary-trees || 342 ||
> || digits-of-e2 || 381 ||
> || circsim || 418 ||
> || S || 451 ||
> || VSM || 497 ||
> || lambda || 617 ||
> || integer || 905 ||
> || n-body || 1043 ||
> || linear || 1070 ||
> || spectral-norm || 1132 ||
> || constraints || 1171 ||
> || exact-reals || 1289 ||
> || mate || 1955 ||
> || fannkuch-redux || 2185 ||
> || k-nucleotide || 2989 ||
> || 17 ||
> || gamteb || 17 ||
> || fft || 18 ||
> || boyer || 20 ||
> || gcd || 20 ||
> || sphere || 21 ||
> || fft2 || 23 ||
> || infer || 25 ||
> || maillist || 25 ||
> || mandel || 27 ||
> || nucleic2 || 30 ||
> || primes || 34 ||
> || cichelli || 35 ||
> || ida || 39 ||
> || listcompr || 41 ||
> || listcopy || 43 ||
> || multiplier || 43 ||
> || wang || 43 ||
> || hpg || 45 ||
> || anna || 46 ||
> || reverse-complem || 46 ||
> || integrate || 48 ||
> || primetest || 50 ||
> || paraffins || 54 ||
> || solid || 54 ||
> || puzzle || 55 ||
> || treejoin || 61 ||
> || compress2 || 64 ||
> || compress || 65 ||
> || event || 65 ||
> || bernouilli || 74 ||
> || comp_lab_zift || 78 ||
> || simple || 86 ||
> || wheel-sieve2 || 91 ||
> || life || 100 ||
> || exp3_8 || 102 ||
> || wave4main || 103 ||
> || typecheck || 106 ||
> || atom || 110 ||
> || fulsom || 112 ||
> || CS || 116 ||
> || para || 120 ||
> || kahan || 138 ||
> || transform || 145 ||
> || power || 150 ||
> || cacheprof || 151 ||
> || hidden || 160 ||
> || scs || 168 ||
> || lcss || 175 ||
> || wheel-sieve1 || 199 ||
> || VS || 204 ||
> || pidigits || 216 ||
> || FS || 222 ||
> || last-piece || 230 ||
> || fasta || 263 ||
> || cryptarithm1 || 264 ||
> || CSD || 272 ||
> || digits-of-e1 || 273 ||
> || binary-trees || 342 ||
> || digits-of-e2 || 381 ||
> || circsim || 418 ||
> || S || 451 ||
> || VSM || 497 ||
> || lambda || 617 ||
> || integer || 905 ||
> || n-body || 1043 ||
> || linear || 1070 ||
> || spectral-norm || 1132 ||
> || constraints || 1171 ||
> || exact-reals || 1289 ||
> || mate || 1955 ||
> || fannkuch-redux || 2185 ||
> || k-nucleotide || 2989 ||
New description:
Currently many benchmarks in a default nofib run have runtimes so low that
they become meaningless.
This comes with a bunch of issues:
* It takes up build/runtime while not actually improving the accuracy of
runtime measurements.
* It skews results as the jump from 1ms to 2ms registers as a 200% change.
* It can hide regression which fall outside of the measured granularity.
* Higher number of runs per benchmark spend far too much time on the few
slower benchmarks.
At the extreme end there are 9 benchmarks with runtimes shorter than the
granularity of 1ms.
For some benchmarks changing this might be as easy as increasing the
number of runs/iterations.
For others changing problem sizes is not as easy since they process more
complex input data.
Below are the runtimes of a recent default nofib run.
||= benchmark =|| Time (ms) ||
|| ansi || 0 ||
|| awards || 0 ||
|| banner || 0 ||
|| calendar || 0 ||
|| expert || 0 ||
|| gen_regexps || 0 ||
|| grep || 0 ||
|| pretty || 0 ||
|| scc || 0 ||
|| cse || 1 ||
|| eliza || 1 ||
|| lift || 1 ||
|| mkhprog || 1 ||
|| prolog || 1 ||
|| sorting || 1 ||
|| veritas || 1 ||
|| knights || 2 ||
|| minimax || 2 ||
|| x2n1 || 2 ||
|| mandel2 || 3 ||
|| parstof || 3 ||
|| fluid || 4 ||
|| pic || 4 ||
|| boyer2 || 5 ||
|| bspt || 5 ||
|| cryptarithm2 || 5 ||
|| fish || 6 ||
|| gg || 6 ||
|| reptile || 6 ||
|| symalg || 6 ||
|| tak || 6 ||
|| VSD || 7 ||
|| rfib || 7 ||
|| queens || 9 ||
|| rewrite || 9 ||
|| sched || 11 ||
|| fem || 12 ||
|| fibheaps || 13 ||
|| parser || 14 ||
|| rsa || 14 ||
|| genfft || 15 ||
|| clausify || 17 ||
|| gamteb || 17 ||
|| fft || 18 ||
|| boyer || 20 ||
|| gcd || 20 ||
|| sphere || 21 ||
|| fft2 || 23 ||
|| infer || 25 ||
|| maillist || 25 ||
|| mandel || 27 ||
|| nucleic2 || 30 ||
|| primes || 34 ||
|| cichelli || 35 ||
|| ida || 39 ||
|| listcompr || 41 ||
|| listcopy || 43 ||
|| multiplier || 43 ||
|| wang || 43 ||
|| hpg || 45 ||
|| anna || 46 ||
|| reverse-complem || 46 ||
|| integrate || 48 ||
|| primetest || 50 ||
|| paraffins || 54 ||
|| solid || 54 ||
|| puzzle || 55 ||
|| treejoin || 61 ||
|| compress2 || 64 ||
|| compress || 65 ||
|| event || 65 ||
|| bernouilli || 74 ||
|| comp_lab_zift || 78 ||
|| simple || 86 ||
|| wheel-sieve2 || 91 ||
|| life || 100 ||
|| exp3_8 || 102 ||
|| wave4main || 103 ||
|| typecheck || 106 ||
|| atom || 110 ||
|| fulsom || 112 ||
|| CS || 116 ||
|| para || 120 ||
|| kahan || 138 ||
|| transform || 145 ||
|| power || 150 ||
|| cacheprof || 151 ||
|| hidden || 160 ||
|| scs || 168 ||
|| lcss || 175 ||
|| wheel-sieve1 || 199 ||
|| VS || 204 ||
|| pidigits || 216 ||
|| FS || 222 ||
|| last-piece || 230 ||
|| fasta || 263 ||
|| cryptarithm1 || 264 ||
|| CSD || 272 ||
|| digits-of-e1 || 273 ||
|| binary-trees || 342 ||
|| digits-of-e2 || 381 ||
|| circsim || 418 ||
|| S || 451 ||
|| VSM || 497 ||
|| lambda || 617 ||
|| integer || 905 ||
|| n-body || 1043 ||
|| linear || 1070 ||
|| spectral-norm || 1132 ||
|| constraints || 1171 ||
|| exact-reals || 1289 ||
|| mate || 1955 ||
|| fannkuch-redux || 2185 ||
|| k-nucleotide || 2989 ||
--
Comment (by AndreasK):
Formatting fix
--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler
More information about the ghc-tickets
mailing list