[GHC] #15357: Make nofib suitable for runtime measurements.
GHC
ghc-devs at haskell.org
Mon Jul 9 16:29:43 UTC 2018
#15357: Make nofib suitable for runtime measurements.
-------------------------------------+-------------------------------------
Reporter: AndreasK | Owner: (none)
Type: task | Status: new
Priority: normal | Milestone: 8.6.1
Component: NoFib | Version: 8.4.3
benchmark suite |
Keywords: | Operating System: Unknown/Multiple
Architecture: | Type of failure: None/Unknown
Unknown/Multiple |
Test Case: | Blocked By:
Blocking: | Related Tickets:
Differential Rev(s): | Wiki Page:
-------------------------------------+-------------------------------------
Currently many benchmarks in a default nofib run have runtimes so low that
they become meaningless.
This comes with a bunch of issues:
* It takes up build/runtime while not actually improving the accuracy of
runtime measurements.
* It skews results as the jump from 1ms to 2ms registers as a 200% change.
* It can hide regression which fall outside of the measured granularity.
* Higher number of runs per benchmark spend far too much time on the few
slower benchmarks.
At the extreme end there are 9 benchmarks with runtimes shorter than the
granularity of 1ms.
For some benchmarks changing this might be as easy as increasing the
number of runs/iterations.
For others changing problem sizes is not as easy since they process more
complex input data.
Below are the runtimes of a recent default nofib run.
||= benchmark =|| Time (ms) ||
|| ansi || 0 ||
|| awards || 0 ||
|| banner || 0 ||
|| calendar || 0 ||
|| expert || 0 ||
|| gen_regexps || 0 ||
|| grep || 0 ||
|| pretty || 0 ||
|| scc || 0 ||
|| cse || 1 ||
|| eliza || 1 ||
|| lift || 1 ||
|| mkhprog || 1 ||
|| prolog || 1 ||
|| sorting || 1 ||
|| veritas || 1 ||
|| knights || 2 ||
|| minimax || 2 ||
|| x2n1 || 2 ||
|| mandel2 || 3 ||
|| parstof || 3 ||
|| fluid || 4 ||
|| pic || 4 ||
|| boyer2 || 5 ||
|| bspt || 5 ||
|| cryptarithm2 || 5 ||
|| fish || 6 ||
|| gg || 6 ||
|| reptile || 6 ||
|| symalg || 6 ||
|| tak || 6 ||
|| VSD || 7 ||
|| rfib || 7 ||
|| queens || 9 ||
|| rewrite || 9 ||
|| sched || 11 ||
|| fem || 12 ||
|| fibheaps || 13 ||
|| parser || 14 ||
|| rsa || 14 ||
|| genfft || 15 ||
|| clausify || ansi || 0 ||
|| awards || 0 ||
|| banner || 0 ||
|| calendar || 0 ||
|| expert || 0 ||
|| gen_regexps || 0 ||
|| grep || 0 ||
|| pretty || 0 ||
|| scc || 0 ||
|| cse || 1 ||
|| eliza || 1 ||
|| lift || 1 ||
|| mkhprog || 1 ||
|| prolog || 1 ||
|| sorting || 1 ||
|| veritas || 1 ||
|| knights || 2 ||
|| minimax || 2 ||
|| x2n1 || 2 ||
|| mandel2 || 3 ||
|| parstof || 3 ||
|| fluid || 4 ||
|| pic || 4 ||
|| boyer2 || 5 ||
|| bspt || 5 ||
|| cryptarithm2 || 5 ||
|| fish || 6 ||
|| gg || 6 ||
|| reptile || 6 ||
|| symalg || 6 ||
|| tak || 6 ||
|| VSD || 7 ||
|| rfib || 7 ||
|| queens || 9 ||
|| rewrite || 9 ||
|| sched || 11 ||
|| fem || 12 ||
|| fibheaps || 13 ||
|| parser || 14 ||
|| rsa || 14 ||
|| genfft || 15 ||
|| clausify || 17 ||
|| gamteb || 17 ||
|| fft || 18 ||
|| boyer || 20 ||
|| gcd || 20 ||
|| sphere || 21 ||
|| fft2 || 23 ||
|| infer || 25 ||
|| maillist || 25 ||
|| mandel || 27 ||
|| nucleic2 || 30 ||
|| primes || 34 ||
|| cichelli || 35 ||
|| ida || 39 ||
|| listcompr || 41 ||
|| listcopy || 43 ||
|| multiplier || 43 ||
|| wang || 43 ||
|| hpg || 45 ||
|| anna || 46 ||
|| reverse-complem || 46 ||
|| integrate || 48 ||
|| primetest || 50 ||
|| paraffins || 54 ||
|| solid || 54 ||
|| puzzle || 55 ||
|| treejoin || 61 ||
|| compress2 || 64 ||
|| compress || 65 ||
|| event || 65 ||
|| bernouilli || 74 ||
|| comp_lab_zift || 78 ||
|| simple || 86 ||
|| wheel-sieve2 || 91 ||
|| life || 100 ||
|| exp3_8 || 102 ||
|| wave4main || 103 ||
|| typecheck || 106 ||
|| atom || 110 ||
|| fulsom || 112 ||
|| CS || 116 ||
|| para || 120 ||
|| kahan || 138 ||
|| transform || 145 ||
|| power || 150 ||
|| cacheprof || 151 ||
|| hidden || 160 ||
|| scs || 168 ||
|| lcss || 175 ||
|| wheel-sieve1 || 199 ||
|| VS || 204 ||
|| pidigits || 216 ||
|| FS || 222 ||
|| last-piece || 230 ||
|| fasta || 263 ||
|| cryptarithm1 || 264 ||
|| CSD || 272 ||
|| digits-of-e1 || 273 ||
|| binary-trees || 342 ||
|| digits-of-e2 || 381 ||
|| circsim || 418 ||
|| S || 451 ||
|| VSM || 497 ||
|| lambda || 617 ||
|| integer || 905 ||
|| n-body || 1043 ||
|| linear || 1070 ||
|| spectral-norm || 1132 ||
|| constraints || 1171 ||
|| exact-reals || 1289 ||
|| mate || 1955 ||
|| fannkuch-redux || 2185 ||
|| k-nucleotide || 2989 ||
|| 17 ||
|| gamteb || 17 ||
|| fft || 18 ||
|| boyer || 20 ||
|| gcd || 20 ||
|| sphere || 21 ||
|| fft2 || 23 ||
|| infer || 25 ||
|| maillist || 25 ||
|| mandel || 27 ||
|| nucleic2 || 30 ||
|| primes || 34 ||
|| cichelli || 35 ||
|| ida || 39 ||
|| listcompr || 41 ||
|| listcopy || 43 ||
|| multiplier || 43 ||
|| wang || 43 ||
|| hpg || 45 ||
|| anna || 46 ||
|| reverse-complem || 46 ||
|| integrate || 48 ||
|| primetest || 50 ||
|| paraffins || 54 ||
|| solid || 54 ||
|| puzzle || 55 ||
|| treejoin || 61 ||
|| compress2 || 64 ||
|| compress || 65 ||
|| event || 65 ||
|| bernouilli || 74 ||
|| comp_lab_zift || 78 ||
|| simple || 86 ||
|| wheel-sieve2 || 91 ||
|| life || 100 ||
|| exp3_8 || 102 ||
|| wave4main || 103 ||
|| typecheck || 106 ||
|| atom || 110 ||
|| fulsom || 112 ||
|| CS || 116 ||
|| para || 120 ||
|| kahan || 138 ||
|| transform || 145 ||
|| power || 150 ||
|| cacheprof || 151 ||
|| hidden || 160 ||
|| scs || 168 ||
|| lcss || 175 ||
|| wheel-sieve1 || 199 ||
|| VS || 204 ||
|| pidigits || 216 ||
|| FS || 222 ||
|| last-piece || 230 ||
|| fasta || 263 ||
|| cryptarithm1 || 264 ||
|| CSD || 272 ||
|| digits-of-e1 || 273 ||
|| binary-trees || 342 ||
|| digits-of-e2 || 381 ||
|| circsim || 418 ||
|| S || 451 ||
|| VSM || 497 ||
|| lambda || 617 ||
|| integer || 905 ||
|| n-body || 1043 ||
|| linear || 1070 ||
|| spectral-norm || 1132 ||
|| constraints || 1171 ||
|| exact-reals || 1289 ||
|| mate || 1955 ||
|| fannkuch-redux || 2185 ||
|| k-nucleotide || 2989 ||
--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler
More information about the ghc-tickets
mailing list