[GHC] #5793: make nofib awesome

[GHC]

#5793: make nofib awesome
--------------------------------------+-------------------------------------
    Reporter:  dterei                 |       Owner:  dterei          
        Type:  task                   |      Status:  new             
    Priority:  normal                 |   Milestone:  _|_             
   Component:  NoFib benchmark suite  |     Version:                  
    Keywords:                         |          Os:  Unknown/Multiple
Architecture:  Unknown/Multiple       |     Failure:  None/Unknown    
  Difficulty:  Unknown                |    Testcase:                  
   Blockedby:                         |    Blocking:  5794            
     Related:                         |  
--------------------------------------+-------------------------------------
Description changed by dterei:

Old description:

> Nofib is the standard tool GHC developers use to benchmark changes to the
> compiler. Its overall design is OK but it's had no love and care for many
> years and has bittrotted such that it isn't useful in a lot of
> situations.
>
> This task is about making nofib useful again.
>
> The breakdown for this is something like:
>
> 1. Think and maybe fix nofib framework design. It has 'ways' which I
> think correspond to compilation method but more in the sense of 'dynamic'
> vs 'static', seems it may not suite being able to use ways for 'fasm' vs
> 'fllvm'. There is also the concept of 'modes' which corresponds to
> different benchmark input. So 'normal' and 'slow' for getting different
> run-times. At moment no easy way to select which benchmark groups to run,
> so may want to change that. I guess we should just decide, what knobs do
> we want to be able to easily tweak, and see how well the current design
> allows that.
>
> 2. Fixup the runtimes for benchmarks to be significant. This might be
> best done by changing the way we run benchmarks and collect results to
> make sure they are meaningful.
>
> E.g., Lots of great discussion and links to papers on this thread
>
> http://www.haskell.org/pipermail/ghc-devs/2013-February/000307.html
>
> 3. Above task is to fix normal but we may want to fixup slow as well and
> perhaps add a 'fast' mode where benchmarks run in around 1 second.
>
> 4. Maybe add more benchmarks to the suite (text, bytestring, performance
> regressions from ghc testsuite, vector....)

New description:

 Nofib is the standard tool GHC developers use to benchmark changes to the
 compiler. Its overall design is OK but it's had no love and care for many
 years and has bittrotted such that it isn't useful in a lot of situations.

 This task is about making nofib useful again.

 The breakdown for this is something like:

 1. Think and maybe fix nofib framework design. It has 'ways' which I think
 correspond to compilation method but more in the sense of 'dynamic' vs
 'static', seems it may not suite being able to use ways for 'fasm' vs
 'fllvm'. There is also the concept of 'modes' which corresponds to
 different benchmark input. So 'normal' and 'slow' for getting different
 run-times. At moment no easy way to select which benchmark groups to run,
 so may want to change that. I guess we should just decide, what knobs do
 we want to be able to easily tweak, and see how well the current design
 allows that.

 '''Note''' there is a shake build system attached that does a lot of this
 (done by Neil Mitchell!). An explanation of it can be found here:
 http://neilmitchell.blogspot.com/2013/02/a-nofib-build-system-using-
 shake.html

 The design discussion of it is mostly lost as it was done on private email
 sorry.

 2. Fixup the runtimes for benchmarks to be significant. This might be best
 done by changing the way we run benchmarks and collect results to make
 sure they are meaningful.

 E.g., Lots of great discussion and links to papers on this thread

 http://www.haskell.org/pipermail/ghc-devs/2013-February/000307.html

 3. Above task is to fix normal but we may want to fixup slow as well and
 perhaps add a 'fast' mode where benchmarks run in around 1 second.

 4. Maybe add more benchmarks to the suite (text, bytestring, performance
 regressions from ghc testsuite, vector....)

--

-- 
Ticket URL: <http://hackage.haskell.org/trac/ghc/ticket/5793#comment:24>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler