[GHC] #9221: (super!) linear slowdown of parallel builds on 40 core machine

[GHC]

#9221: (super!) linear slowdown of parallel builds on 40 core machine
-------------------------------------+-------------------------------------
              Reporter:  carter      |            Owner:
                  Type:  bug         |           Status:  new
              Priority:  high        |        Milestone:  7.10.1
             Component:  Compiler    |          Version:  7.8.2
            Resolution:              |         Keywords:
      Operating System:              |     Architecture:  Unknown/Multiple
  Unknown/Multiple                   |       Difficulty:  Unknown
       Type of failure:  Compile-    |       Blocked By:
  time performance bug               |  Related Tickets:  #910
             Test Case:              |
              Blocking:              |
Differential Revisions:              |
-------------------------------------+-------------------------------------

Comment (by gintas):

 I ran some experiments with -A and it does help a lot with performance,
 but also increases peak memory usage. I observed continuous improvement
 all the way from -A1m to -A128m in terms of walltime (41s to 36s), but
 "total memory in use" also went up from 265MB to 2182MB. Not sure where
 the sweet spot is.

 -A seems to help especially if the number of capabilities exceeds the
 number of cores. With 32 capabilities on a 16 core machine, a -qg run took
 50s, -A128m took 41s (still a penalty over 36s but not nearly as bad) and
 a vanilla run took almost 2min. Of course, total memory use with -A128m
 went up to 4388m...

 Looks like the sweet spot is somewhere in the middle. How about building
 ghc --with-rtsopts=-A16m by default? Are there any downsides to that? On
 my machine the "total memory in use" overhead for that is less than 2x
 (475MB vs 265MB) and it really helps especially in the degenerate cases.

 (The machine I was testing on had 16 real cores without any
 hyperthreading.)

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/9221#comment:14>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler