[GHC] #5793: make nofib awesome
GHC
ghc-devs at haskell.org
Wed Jan 9 17:34:37 UTC 2019
#5793: make nofib awesome
-------------------------------------+-------------------------------------
Reporter: dterei | Owner: michalt
Type: task | Status: new
Priority: normal | Milestone: ⊥
Component: NoFib benchmark | Version:
suite |
Resolution: | Keywords:
Operating System: Unknown/Multiple | Architecture:
| Unknown/Multiple
Type of failure: None/Unknown | Test Case:
Blocked By: 9571 | Blocking:
Related Tickets: #15357 #15333 | Differential Rev(s):
#15999 |
Wiki Page: |
-------------------------------------+-------------------------------------
Comment (by Sebastian Graf <sebastian.graf@…>):
In [changeset:"8632268ad8405f0c01aaad3ad16e23c65771ba49/nofib"
8632268/nofib]:
{{{
#!CommitTicketReference repository="nofib"
revision="8632268ad8405f0c01aaad3ad16e23c65771ba49"
Stabilise benchmarks wrt. GC
Summary:
This is due to #15999, a follow-up on #5793 and #15357 and changes all
benchmarks, some of them (i.e. `wheel-sieve1`, `awards`) rather
drastically.
The general plan is outlined in #15999: Identify GC-sensitive benchmarks
by
looking at how productivity rates change over different nursery sizes and
iterate `main` of these benchmarks often enough for the non-monotony and
discontinuities to go away.
I was paying attention that the benchmarked logic is actually run $n times
more
often, rather than just benchmarking IO operations printing the result of
CAFs.
When I found benchmarks with insignificant runtime (#15357), I made sure
that
parameters/input files were adjusted so that runtime of the different
modes
fall within the ranges proposed in
https://ghc.haskell.org/trac/ghc/ticket/15357#comment:4
- fast: 0.1-0.2s
- norm: 1-2s
- slow: 5-10s
This is what I did:
- Stabilise bernoulli
- Stabilise digits-of-e1
- Stabilise digits-of-e2
- Stabilise gen_regexp
- Adjust running time of integrate
- Adjust running time of kahan
- Stabilise paraffins
- Stabilise primes
- Adjust running time of rfib
- Adjust running time of tak
- Stabilise wheel-sieve1
- Stabilise wheel-sieve2
- Adjust running time of x2n1
- Adjust running time of ansi
- Adjust running time of atom
- Make awards benchmark something other than IO
- Adjust running time of banner
- Stabilise boyer
- Adjust running time of boyer2
- Adjust running time of queens
- Adjust running time of calendar
- Adjust runtime of cichelli
- Stabilise circsim
- Stabilise clausify
- Stabilise constraints with moderate success
- Adjust running time of cryptarithm1
- Adjust running time of cryptarythm2
- Adjust running time of cse
- Adjust running time of eliza
- Adjust running time of exact-reals
- Adjust running time of expert
- Stabilise fft2
- Stabilise fibheaps
- Stabilise fish
- Adjust running time for gcd
- Stabilise comp_lab_zift
- Stabilise event
- Stabilise fft
- Stabilise genfft
- Stabilise ida
- Adjust running time for listcompr
- Adjust running time for listcopy
- Adjust running time of nucleic2
- Attempt to stabilise parstof
- Stabilise sched
- Stabilise solid
- Adjust running time of transform
- Adjust running time of typecheck
- Stabilise wang
- Stabilise wave4main
- Adjust running time of integer
- Adjust running time of knights
- Stabilise lambda
- Stabilise lcss
- Stabilise life
- Stabilise mandel
- Stabilise mandel2
- Adjust running time of mate
- Stabilise minimax
- Adjust running time of multiplier
- Adjust running time of para
- Stabilise power
- Adjust running time of primetest
- Stabilise puzzle with mild success
- Adjust running time for rewrite
- Stabilise simple with mild success
- Stabilise sorting
- Stabilise sphere
- Stabilise treejoin
- Stabilise anna
- Stabilise bspt
- Stabilise cacheprof
- Stablise compress
- Stablise compress2
- Stabilise fem
- Adjust running time of fluid
- Stabilise fulsom
- Stabilise gamteb
- Stabilise gg
- Stabilise grep
- Adjust running time of hidden
- Stabilise hpg
- Stabilise infer
- Stabilise lift
- Stabilise linear
- Attempt to stabilise maillist
- Stabilise mkhprog
- Stabilise parser
- Stabilise pic
- Stabilise prolog
- Attempt to stabilise reptile
- Adjust running time of rsa
- Adjust running time of scs
- Stabilise symalg
- Stabilise veritas
- Stabilise binary-trees
- Adjust running time of fasta
- Adjust running time of k-nucleotide
- Adjust running time of pidigits
- Adjust running time of reverse-complement
- Adjust running time of spectral-norm
- Adjust running time of fannkuch-redux
- Adjust running time for n-body
Problematic benchmarks:
- `last-piece`: Unclear how to stabilise. Runs for 300ms and I can't make
up smaller inputs because I don't understand what it does.
- `pretty`: It's just much too small to be relevant at all. Maybe we want
to get rid of this one?
- `scc`: Same as `pretty`. The input graph for which SCC analysis is done
is much too small and I can't find good directed example graphs on the
internet.
- `secretary`: Apparently this needs `-package random` and consequently
hasn't been run for a long time.
- `simple`: Same as `last-piece`. Decent runtime (70ms), but it's unstable
and I see no way to iterate it ~100 times in fast mode.
- `eff`: Every benchmark is problematic here. Not from the point of view
of allocations, but because the actual logic is vacuous. IMO, these should
be performance tests, not actual benchmarks. Alternatively, write an
actual application that makes use of algebraic effects.
- `maillist`: Too trivial. It's just String/list manipulation, not
representative of any Haskell code we would write today (no use of base
library functions which could be fused, uses String instead of Text). It's
only 75 loc according to `cloc`, that's not a `real` application.
Reviewers: simonpj, simonmar, bgamari, AndreasK, osa1, alpmestan, O26
nofib
GHC Trac Issues: #15999
Differential Revision: https://phabricator.haskell.org/D5438
}}}
--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/5793#comment:41>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler
More information about the ghc-tickets
mailing list