[GHC] #15357: Make nofib suitable for runtime measurements.

GHC ghc-devs at haskell.org
Mon Jul 9 16:31:58 UTC 2018


#15357: Make nofib suitable for runtime measurements.
-------------------------------------+-------------------------------------
        Reporter:  AndreasK          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.6.1
       Component:  NoFib benchmark   |              Version:  8.4.3
  suite                              |
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Old description:

> Currently many benchmarks in a default nofib run have runtimes so low
> that they become meaningless.
>
> This comes with a bunch of issues:
> * It takes up build/runtime while not actually improving the accuracy of
> runtime measurements.
> * It skews results as the jump from 1ms to 2ms registers as a 200%
> change.
> * It can hide regression which fall outside of the measured granularity.
> * Higher number of runs per benchmark spend far too much time on the few
> slower benchmarks.
>
> At the extreme end there are 9 benchmarks with runtimes shorter than the
> granularity of 1ms.
>
> For some benchmarks changing this might be as easy as increasing the
> number of runs/iterations.
> For others changing problem sizes is not as easy since they process more
> complex input data.
>
> Below are the runtimes of a recent default nofib run.
>

> ||= benchmark =|| Time (ms) ||
> ||      ansi    ||      0       ||
> ||      awards  ||      0       ||
> ||      banner  ||      0       ||
> ||      calendar        ||      0       ||
> ||      expert  ||      0       ||
> ||      gen_regexps     ||      0       ||
> ||      grep    ||      0       ||
> ||      pretty  ||      0       ||
> ||      scc     ||      0       ||
> ||      cse     ||      1       ||
> ||      eliza   ||      1       ||
> ||      lift    ||      1       ||
> ||      mkhprog ||      1       ||
> ||      prolog  ||      1       ||
> ||      sorting ||      1       ||
> ||      veritas ||      1       ||
> ||      knights ||      2       ||
> ||      minimax ||      2       ||
> ||      x2n1    ||      2       ||
> ||      mandel2 ||      3       ||
> ||      parstof ||      3       ||
> ||      fluid   ||      4       ||
> ||      pic     ||      4       ||
> ||      boyer2  ||      5       ||
> ||      bspt    ||      5       ||
> ||      cryptarithm2    ||      5       ||
> ||      fish    ||      6       ||
> ||      gg      ||      6       ||
> ||      reptile ||      6       ||
> ||      symalg  ||      6       ||
> ||      tak     ||      6       ||
> ||      VSD     ||      7       ||
> ||      rfib    ||      7       ||
> ||      queens  ||      9       ||
> ||      rewrite ||      9       ||
> ||      sched   ||      11      ||
> ||      fem     ||      12      ||
> ||      fibheaps        ||      13      ||
> ||      parser  ||      14      ||
> ||      rsa     ||      14      ||
> ||      genfft  ||      15      ||
> ||      clausify        ||      ansi    ||      0       ||
> ||      awards  ||      0       ||
> ||      banner  ||      0       ||
> ||      calendar        ||      0       ||
> ||      expert  ||      0       ||
> ||      gen_regexps     ||      0       ||
> ||      grep    ||      0       ||
> ||      pretty  ||      0       ||
> ||      scc     ||      0       ||
> ||      cse     ||      1       ||
> ||      eliza   ||      1       ||
> ||      lift    ||      1       ||
> ||      mkhprog ||      1       ||
> ||      prolog  ||      1       ||
> ||      sorting ||      1       ||
> ||      veritas ||      1       ||
> ||      knights ||      2       ||
> ||      minimax ||      2       ||
> ||      x2n1    ||      2       ||
> ||      mandel2 ||      3       ||
> ||      parstof ||      3       ||
> ||      fluid   ||      4       ||
> ||      pic     ||      4       ||
> ||      boyer2  ||      5       ||
> ||      bspt    ||      5       ||
> ||      cryptarithm2    ||      5       ||
> ||      fish    ||      6       ||
> ||      gg      ||      6       ||
> ||      reptile ||      6       ||
> ||      symalg  ||      6       ||
> ||      tak     ||      6       ||
> ||      VSD     ||      7       ||
> ||      rfib    ||      7       ||
> ||      queens  ||      9       ||
> ||      rewrite ||      9       ||
> ||      sched   ||      11      ||
> ||      fem     ||      12      ||
> ||      fibheaps        ||      13      ||
> ||      parser  ||      14      ||
> ||      rsa     ||      14      ||
> ||      genfft  ||      15      ||
> ||      clausify        ||      17      ||
> ||      gamteb  ||      17      ||
> ||      fft     ||      18      ||
> ||      boyer   ||      20      ||
> ||      gcd     ||      20      ||
> ||      sphere  ||      21      ||
> ||      fft2    ||      23      ||
> ||      infer   ||      25      ||
> ||      maillist        ||      25      ||
> ||      mandel  ||      27      ||
> ||      nucleic2        ||      30      ||
> ||      primes  ||      34      ||
> ||      cichelli        ||      35      ||
> ||      ida     ||      39      ||
> ||      listcompr       ||      41      ||
> ||      listcopy        ||      43      ||
> ||      multiplier      ||      43      ||
> ||      wang    ||      43      ||
> ||      hpg     ||      45      ||
> ||      anna    ||      46      ||
> ||      reverse-complem ||      46      ||
> ||      integrate       ||      48      ||
> ||      primetest       ||      50      ||
> ||      paraffins       ||      54      ||
> ||      solid   ||      54      ||
> ||      puzzle  ||      55      ||
> ||      treejoin        ||      61      ||
> ||      compress2       ||      64      ||
> ||      compress        ||      65      ||
> ||      event   ||      65      ||
> ||      bernouilli      ||      74      ||
> ||      comp_lab_zift   ||      78      ||
> ||      simple  ||      86      ||
> ||      wheel-sieve2    ||      91      ||
> ||      life    ||      100     ||
> ||      exp3_8  ||      102     ||
> ||      wave4main       ||      103     ||
> ||      typecheck       ||      106     ||
> ||      atom    ||      110     ||
> ||      fulsom  ||      112     ||
> ||      CS      ||      116     ||
> ||      para    ||      120     ||
> ||      kahan   ||      138     ||
> ||      transform       ||      145     ||
> ||      power   ||      150     ||
> ||      cacheprof       ||      151     ||
> ||      hidden  ||      160     ||
> ||      scs     ||      168     ||
> ||      lcss    ||      175     ||
> ||      wheel-sieve1    ||      199     ||
> ||      VS      ||      204     ||
> ||      pidigits        ||      216     ||
> ||      FS      ||      222     ||
> ||      last-piece      ||      230     ||
> ||      fasta   ||      263     ||
> ||      cryptarithm1    ||      264     ||
> ||      CSD     ||      272     ||
> ||      digits-of-e1    ||      273     ||
> ||      binary-trees    ||      342     ||
> ||      digits-of-e2    ||      381     ||
> ||      circsim ||      418     ||
> ||      S       ||      451     ||
> ||      VSM     ||      497     ||
> ||      lambda  ||      617     ||
> ||      integer ||      905     ||
> ||      n-body  ||      1043    ||
> ||      linear  ||      1070    ||
> ||      spectral-norm   ||      1132    ||
> ||      constraints     ||      1171    ||
> ||      exact-reals     ||      1289    ||
> ||      mate    ||      1955    ||
> ||      fannkuch-redux  ||      2185    ||
> ||      k-nucleotide    ||      2989    ||
> ||      17      ||
> ||      gamteb  ||      17      ||
> ||      fft     ||      18      ||
> ||      boyer   ||      20      ||
> ||      gcd     ||      20      ||
> ||      sphere  ||      21      ||
> ||      fft2    ||      23      ||
> ||      infer   ||      25      ||
> ||      maillist        ||      25      ||
> ||      mandel  ||      27      ||
> ||      nucleic2        ||      30      ||
> ||      primes  ||      34      ||
> ||      cichelli        ||      35      ||
> ||      ida     ||      39      ||
> ||      listcompr       ||      41      ||
> ||      listcopy        ||      43      ||
> ||      multiplier      ||      43      ||
> ||      wang    ||      43      ||
> ||      hpg     ||      45      ||
> ||      anna    ||      46      ||
> ||      reverse-complem ||      46      ||
> ||      integrate       ||      48      ||
> ||      primetest       ||      50      ||
> ||      paraffins       ||      54      ||
> ||      solid   ||      54      ||
> ||      puzzle  ||      55      ||
> ||      treejoin        ||      61      ||
> ||      compress2       ||      64      ||
> ||      compress        ||      65      ||
> ||      event   ||      65      ||
> ||      bernouilli      ||      74      ||
> ||      comp_lab_zift   ||      78      ||
> ||      simple  ||      86      ||
> ||      wheel-sieve2    ||      91      ||
> ||      life    ||      100     ||
> ||      exp3_8  ||      102     ||
> ||      wave4main       ||      103     ||
> ||      typecheck       ||      106     ||
> ||      atom    ||      110     ||
> ||      fulsom  ||      112     ||
> ||      CS      ||      116     ||
> ||      para    ||      120     ||
> ||      kahan   ||      138     ||
> ||      transform       ||      145     ||
> ||      power   ||      150     ||
> ||      cacheprof       ||      151     ||
> ||      hidden  ||      160     ||
> ||      scs     ||      168     ||
> ||      lcss    ||      175     ||
> ||      wheel-sieve1    ||      199     ||
> ||      VS      ||      204     ||
> ||      pidigits        ||      216     ||
> ||      FS      ||      222     ||
> ||      last-piece      ||      230     ||
> ||      fasta   ||      263     ||
> ||      cryptarithm1    ||      264     ||
> ||      CSD     ||      272     ||
> ||      digits-of-e1    ||      273     ||
> ||      binary-trees    ||      342     ||
> ||      digits-of-e2    ||      381     ||
> ||      circsim ||      418     ||
> ||      S       ||      451     ||
> ||      VSM     ||      497     ||
> ||      lambda  ||      617     ||
> ||      integer ||      905     ||
> ||      n-body  ||      1043    ||
> ||      linear  ||      1070    ||
> ||      spectral-norm   ||      1132    ||
> ||      constraints     ||      1171    ||
> ||      exact-reals     ||      1289    ||
> ||      mate    ||      1955    ||
> ||      fannkuch-redux  ||      2185    ||
> ||      k-nucleotide    ||      2989    ||

New description:

 Currently many benchmarks in a default nofib run have runtimes so low that
 they become meaningless.

 This comes with a bunch of issues:
 * It takes up build/runtime while not actually improving the accuracy of
 runtime measurements.
 * It skews results as the jump from 1ms to 2ms registers as a 200% change.
 * It can hide regression which fall outside of the measured granularity.
 * Higher number of runs per benchmark spend far too much time on the few
 slower benchmarks.

 At the extreme end there are 9 benchmarks with runtimes shorter than the
 granularity of 1ms.

 For some benchmarks changing this might be as easy as increasing the
 number of runs/iterations.
 For others changing problem sizes is not as easy since they process more
 complex input data.

 Below are the runtimes of a recent default nofib run.


 ||= benchmark =|| Time (ms) ||
 ||      ansi    ||      0       ||
 ||      awards  ||      0       ||
 ||      banner  ||      0       ||
 ||      calendar        ||      0       ||
 ||      expert  ||      0       ||
 ||      gen_regexps     ||      0       ||
 ||      grep    ||      0       ||
 ||      pretty  ||      0       ||
 ||      scc     ||      0       ||
 ||      cse     ||      1       ||
 ||      eliza   ||      1       ||
 ||      lift    ||      1       ||
 ||      mkhprog ||      1       ||
 ||      prolog  ||      1       ||
 ||      sorting ||      1       ||
 ||      veritas ||      1       ||
 ||      knights ||      2       ||
 ||      minimax ||      2       ||
 ||      x2n1    ||      2       ||
 ||      mandel2 ||      3       ||
 ||      parstof ||      3       ||
 ||      fluid   ||      4       ||
 ||      pic     ||      4       ||
 ||      boyer2  ||      5       ||
 ||      bspt    ||      5       ||
 ||      cryptarithm2    ||      5       ||
 ||      fish    ||      6       ||
 ||      gg      ||      6       ||
 ||      reptile ||      6       ||
 ||      symalg  ||      6       ||
 ||      tak     ||      6       ||
 ||      VSD     ||      7       ||
 ||      rfib    ||      7       ||
 ||      queens  ||      9       ||
 ||      rewrite ||      9       ||
 ||      sched   ||      11      ||
 ||      fem     ||      12      ||
 ||      fibheaps        ||      13      ||
 ||      parser  ||      14      ||
 ||      rsa     ||      14      ||
 ||      genfft  ||      15      ||
 ||      clausify        ||      17      ||
 ||      gamteb  ||      17      ||
 ||      fft     ||      18      ||
 ||      boyer   ||      20      ||
 ||      gcd     ||      20      ||
 ||      sphere  ||      21      ||
 ||      fft2    ||      23      ||
 ||      infer   ||      25      ||
 ||      maillist        ||      25      ||
 ||      mandel  ||      27      ||
 ||      nucleic2        ||      30      ||
 ||      primes  ||      34      ||
 ||      cichelli        ||      35      ||
 ||      ida     ||      39      ||
 ||      listcompr       ||      41      ||
 ||      listcopy        ||      43      ||
 ||      multiplier      ||      43      ||
 ||      wang    ||      43      ||
 ||      hpg     ||      45      ||
 ||      anna    ||      46      ||
 ||      reverse-complem ||      46      ||
 ||      integrate       ||      48      ||
 ||      primetest       ||      50      ||
 ||      paraffins       ||      54      ||
 ||      solid   ||      54      ||
 ||      puzzle  ||      55      ||
 ||      treejoin        ||      61      ||
 ||      compress2       ||      64      ||
 ||      compress        ||      65      ||
 ||      event   ||      65      ||
 ||      bernouilli      ||      74      ||
 ||      comp_lab_zift   ||      78      ||
 ||      simple  ||      86      ||
 ||      wheel-sieve2    ||      91      ||
 ||      life    ||      100     ||
 ||      exp3_8  ||      102     ||
 ||      wave4main       ||      103     ||
 ||      typecheck       ||      106     ||
 ||      atom    ||      110     ||
 ||      fulsom  ||      112     ||
 ||      CS      ||      116     ||
 ||      para    ||      120     ||
 ||      kahan   ||      138     ||
 ||      transform       ||      145     ||
 ||      power   ||      150     ||
 ||      cacheprof       ||      151     ||
 ||      hidden  ||      160     ||
 ||      scs     ||      168     ||
 ||      lcss    ||      175     ||
 ||      wheel-sieve1    ||      199     ||
 ||      VS      ||      204     ||
 ||      pidigits        ||      216     ||
 ||      FS      ||      222     ||
 ||      last-piece      ||      230     ||
 ||      fasta   ||      263     ||
 ||      cryptarithm1    ||      264     ||
 ||      CSD     ||      272     ||
 ||      digits-of-e1    ||      273     ||
 ||      binary-trees    ||      342     ||
 ||      digits-of-e2    ||      381     ||
 ||      circsim ||      418     ||
 ||      S       ||      451     ||
 ||      VSM     ||      497     ||
 ||      lambda  ||      617     ||
 ||      integer ||      905     ||
 ||      n-body  ||      1043    ||
 ||      linear  ||      1070    ||
 ||      spectral-norm   ||      1132    ||
 ||      constraints     ||      1171    ||
 ||      exact-reals     ||      1289    ||
 ||      mate    ||      1955    ||
 ||      fannkuch-redux  ||      2185    ||
 ||      k-nucleotide    ||      2989    ||

--

Comment (by AndreasK):

 Formatting fix

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:1>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler


More information about the ghc-tickets mailing list