[commit: testsuite] master: Higher residency in Haddock (15c09a3)

[Simon Peyton-Jones]

I agree.  The most direct approach is for someone to build profiled compilers and get some sense of where the space and time is going in a run of Haddock, or GHC.  

I would be thrilled if someone felt able to do some forensic performance comparison between different versions of GHC and/or Haddock. Would anyone like to do that?  There is almost certainly plenty of low-hanging fruit -- you might speed up GHC by 20% and everyone would be Very Happy.

On particular compilation of the Haddock tests is that they embody two distinct sources of variation: 
  1.  Cabal (the source code examined by the run of Haddock
      has more/different code
  2.  Haddock itself may have gotten slower/fatter

(1) is a motivation-sapping problem.  Performance tests should have stable input data.

Better tools would certainly help. If we are to retain memory-size numbers, maybe the tests should always be run in such a way that the numbers are stable and predictable.

The other real difficulty is this: if space usage really does go up by 10% in an extremely large and complicated program like Haddock or GHC, it can be jolly hard to nail down what's causing it.  If it goes through the roof, it's (relatively) easy to find.  If bytes-allocated goes up it's easy to find (use -ticky). But a 10% increase in space usage is always going be hard.

Simon



| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Simon
| Marlow
| Sent: 28 November 2013 12:31
| To: ghc-devs at haskell.org
| Subject: Re: [commit: testsuite] master: Higher residency in Haddock
| (15c09a3)
| 
| On 22/11/13 16:43, git at git.haskell.org wrote:
| > Repository : ssh://git@git.haskell.org/testsuite
| >
| > On branch  : master
| > Link       :
| http://ghc.haskell.org/trac/ghc/changeset/15c09a332cd24e5d6c4c9a0ede9b1
| 51c9a095f66/testsuite
| >
| >> ---------------------------------------------------------------
| >
| > commit 15c09a332cd24e5d6c4c9a0ede9b151c9a095f66
| > Author: Simon Peyton Jones <simonpj at microsoft.com>
| > Date:   Fri Nov 22 16:42:43 2013 +0000
| >
| >      Higher residency in Haddock
| >
| >      I think there really is a slight worsening in the situation
| here, but
| >      it needs someone to build a profiled compiler and take a proper
| look.
| 
| Maybe make a ticket for the next milestone then?
| 
| I'm getting slightly worried that we keep pushing these performance
| bounds up without really investigating why.  The great thing about the
| perf tests is that they catch something immediately, rather than us
| having to bisect it or do lengthy profiling investigations later.
| 
| What should we do about this?  I realise it's a pain when you've got a
| working patch and the only thing holding it up is some random perf test
| that seems to be completely unrelated.  Let's have a brainstorm on how
| we can improve things.  What tools can we build to help?  Maybe this is
| a good way that new contributors could get involved.
| 
| Here's one idea off the top of my head: let's make a way to take heap
| samples at deterministic points during compilation, say between phases.
|   Then make a tool to compare profiles made this way, so that we can
| say
| what changed between two compiler versions.  I'm pretty sure this will
| spot leaks pretty quickly, and tell you something about what is leaking
| (which constructors).
| 
| I think we've accrued some bloat recently.  I had to increase the
| memory
| in my VM because it ceased to be able to build HEAD at some point.
| 
| Cheers,
| Simon
| 
| _______________________________________________
| ghc-devs mailing list
| ghc-devs at haskell.org
| http://www.haskell.org/mailman/listinfo/ghc-devs