[commit: testsuite] master: Higher residency in Haddock (15c09a3)

[Simon Marlow]

On 28/11/13 12:56, Simon Peyton-Jones wrote:
> I agree.  The most direct approach is for someone to build profiled compilers and get some sense of where the space and time is going in a run of Haddock, or GHC.
>
> I would be thrilled if someone felt able to do some forensic performance comparison between different versions of GHC and/or Haddock. Would anyone like to do that?  There is almost certainly plenty of low-hanging fruit -- you might speed up GHC by 20% and everyone would be Very Happy.
>
> On particular compilation of the Haddock tests is that they embody two distinct sources of variation:
>    1.  Cabal (the source code examined by the run of Haddock
>        has more/different code
>    2.  Haddock itself may have gotten slower/fatter

Most of the cost in Haddock is in typechecking all the source code, 
which is done by GHC.  I think this is why we have perf tests for 
Haddock - they're GHC tests, kind-of.  The annoying thing about the 
Haddock perf tests is that they're difficult to re-run though.

> (1) is a motivation-sapping problem.  Performance tests should have stable input data.

Yes, that's not good.  Cabal changing is certainly a legitimate reason 
to bump the thresholds for the Haddock tests though.

> Better tools would certainly help. If we are to retain memory-size numbers, maybe the tests should always be run in such a way that the numbers are stable and predictable.

I've been thinking about this.  Just increasing the frequency of samples 
would help, but I think taking samples at deterministic points (like 
between phases) would help more.  It needs someone to add some support 
for this though, in the form of an API for heap sampling that we can 
call from Haskell.

Cheers,
Simon


> Simon
>
>
>
> | -----Original Message-----
> | From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Simon
> | Marlow
> | Sent: 28 November 2013 12:31
> | To: ghc-devs at haskell.org
> | Subject: Re: [commit: testsuite] master: Higher residency in Haddock
> | (15c09a3)
> |
> | On 22/11/13 16:43, git at git.haskell.org wrote:
> | > Repository : ssh://git@git.haskell.org/testsuite
> | >
> | > On branch  : master
> | > Link       :
> | http://ghc.haskell.org/trac/ghc/changeset/15c09a332cd24e5d6c4c9a0ede9b1
> | 51c9a095f66/testsuite
> | >
> | >> ---------------------------------------------------------------
> | >
> | > commit 15c09a332cd24e5d6c4c9a0ede9b151c9a095f66
> | > Author: Simon Peyton Jones <simonpj at microsoft.com>
> | > Date:   Fri Nov 22 16:42:43 2013 +0000
> | >
> | >      Higher residency in Haddock
> | >
> | >      I think there really is a slight worsening in the situation
> | here, but
> | >      it needs someone to build a profiled compiler and take a proper
> | look.
> |
> | Maybe make a ticket for the next milestone then?
> |
> | I'm getting slightly worried that we keep pushing these performance
> | bounds up without really investigating why.  The great thing about the
> | perf tests is that they catch something immediately, rather than us
> | having to bisect it or do lengthy profiling investigations later.
> |
> | What should we do about this?  I realise it's a pain when you've got a
> | working patch and the only thing holding it up is some random perf test
> | that seems to be completely unrelated.  Let's have a brainstorm on how
> | we can improve things.  What tools can we build to help?  Maybe this is
> | a good way that new contributors could get involved.
> |
> | Here's one idea off the top of my head: let's make a way to take heap
> | samples at deterministic points during compilation, say between phases.
> |   Then make a tool to compare profiles made this way, so that we can
> | say
> | what changed between two compiler versions.  I'm pretty sure this will
> | spot leaks pretty quickly, and tell you something about what is leaking
> | (which constructors).
> |
> | I think we've accrued some bloat recently.  I had to increase the
> | memory
> | in my VM because it ceased to be able to build HEAD at some point.
> |
> | Cheers,
> | Simon
> |
> | _______________________________________________
> | ghc-devs mailing list
> | ghc-devs at haskell.org
> | http://www.haskell.org/mailman/listinfo/ghc-devs
>