[GHC] #910: --make should have a -j flag for parallel building

[GHC]

#910: --make should have a -j flag for parallel building
-------------------------------------+------------------------------------
        Reporter:  igloo             |            Owner:
            Type:  feature request   |           Status:  patch
        Priority:  normal            |        Milestone:  _|_
       Component:  Compiler          |          Version:  6.4.2
      Resolution:                    |         Keywords:
Operating System:  Unknown/Multiple  |     Architecture:  Unknown/Multiple
 Type of failure:  None/Unknown      |       Difficulty:  Unknown
       Test Case:  N/A               |       Blocked By:
        Blocking:                    |  Related Tickets:
-------------------------------------+------------------------------------
Changes (by parcs):

 * status:  new => patch


Comment:

 I've been working on this for a while, including this year's GSoC. Since
 the GHC 7.8 feature freeze is imminent, I would like to post a stable
 subset of my current progress for review and possible inclusion to GHC
 7.8. This subset of changes includes the bare minimum required to build
 multiple modules in parallel, and the parallel upsweep itself. Each patch
 is for the most part self-explanatory, I hope.

 https://github.com/parcs/ghc/commits/ghc-parmake-gsoc

 The speedups provided by the parallel upsweep are decent: I can realize a
 1.8x speedup when compiling the Cabal library with `-j3 -O2`, and a 2.4x
 speedup when compiling 7 independent, relatively large modules with `-j3
 -O0`, for example. The performance/thread ratio seems to peak at -j3
 (which instructs the runtime to use 3 capabilities).

 The performance of the sequential upsweep is not significantly impacted by
 these patches: compiling the Cabal library with `-O2` takes about 1%
 longer with the patches than without.

 I have not tested these patches on any platform other than x86_64/Linux,
 but I have no reason to believe that behavior would differ on other
 platforms. Nonetheless, testing on other platforms is necessary and
 appreciated.

 These changes are well-tested and stable, but there is a single bug that
 escapes me which can be triggered by the testsuite. When running the
 testsuite with e.g. `EXTRA_HC_OPTS=-j2`, there is about a 1/1000 chance
 that a compiler process will exit with

 {{{
 ghc-stage2: GHC.Event.Manager.loop: state is already Finished
 }}}

 On a separate machine, I don't ever trigger the aforementioned bug, but
 instead the compiler process on rare occasion never exits. The testsuite
 script eventually kills the process, causing the particular test to fail
 with 'exit code non-0'.

 These are likely bugs in the IO manager triggered by the changing of the
 number of capabilities at runtime. If I instead explicitly set the number
 of capabilities at startup with `EXTRA_HC_OPTS=-j2 +RTS -N2 -RTS` then
 neither bug manifests. I don't yet understand the IO manager well enough
 to fix this issue though.

 Other than that though, this feature Just Works. One simply has to pass
 the `-jN` flag to GHC and the build will just finish faster.

 Questions or comments? I have likely not explained everything I ought to
 explain.

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/910#comment:22>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler