[GHC] #15357: Make nofib suitable for runtime measurements.

[GHC]

#15357: Make nofib suitable for runtime measurements.
-------------------------------------+-------------------------------------
        Reporter:  AndreasK          |                Owner:  (none)
            Type:  task              |               Status:  new
        Priority:  normal            |            Milestone:  8.6.1
       Component:  NoFib benchmark   |              Version:  8.4.3
  suite                              |
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:                    |  Differential Rev(s):
       Wiki Page:                    |
-------------------------------------+-------------------------------------

Comment (by AndreasK):

 > Are you offering to help?! It would be great if so.

 I'm currently working on the NCG for Summer of Code so not before that is
 over.

 I might get around to it after, or I might never so if someone else picks
 this up I would be more than grateful.

 > there's already FAST/NORM/SLOW setting that is supposed to choose per-
 benchmarks configurations that take 1s/5s/20s or something vaguely like
 that.

 Makefiles can set different parameters for the different settings. While
 helpful to main issue is that:
 * There are a benchmarks which use the same setting for all three
 variants.
 * It doesn't help with keeping runtimes of different benchmarks in the
 same ballpark.

 Ideally we would try to balance runtime so that for example:
 * Fast (0.1s-1s) - Enough for allocation/instruction counts or rough
 runtime measurements.
 * normal (1s-2s) - Decent runtime measurements
 * slow (5s-10s) - More precision.

 Currently runtimes for slow just in shootout vary from 0.5s to 40s.
 So if we only run these two benchmarks we split benchmark time <2%:>98%.
 Not exactly a good use of resources.


 One side effect of adjusting runtimes would be that it would invalidate
 any long term performance comparison.
 So I'm also not sure if such a patch would be uncontroversial.

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/15357#comment:4>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler