How to better parallelize GHC build.

[Simon Peyton Jones]

As Neil says, I'm hoping that the new Shake-based build system will make a big difference. It's not certain that we'll switch to it, but I very much hope that we will.  Fortunately we can work it side-by-side with the old system, so I hope it'll just be a question of switching over because it is manifestly better.

Andrey can give a status report.

Simon

| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Karel
| Gardas
| Sent: 07 March 2015 10:50
| To: ghc-devs
| Subject: How to better parallelize GHC build.
| 
| 
| Folks,
| 
| first of all, I remember someone already mentioned issue with decreased
| parallelism of the GHC build recently somewhere but I cann't find it
| now. Sorry, for that since otherwise I would use this thread if it was
| on this mailing list.
| 
| Anyway, while working on SPARC NCG I'm using T2000 which provides 32
| threads/8 core UltraSPARC T1 CPU. The property of this machine is that
| it's really slow on single-threaded work. To squeeze some perf from it
| man really needs to push 32 threads of work on it. Now, it really hurts
| my nerves to see it's lazy building/running just one or two ghc
| processes. To verify the fact I've created simple script to collect
| number of ghc processes over time and putting this to graph. The result
| is in the attached picture. The graph is result of running:
| 
| gmake -j64
| 
| anyway, the average number of running ghc processes is 4.4 and the
| median value is 2. IMHO such low number not only hurts build times on
| something like CMT SPARC machine, but also on let say a cluster of ARM
| machines using NFS and also on common engineering workstations which
| provide these days (IMHO!) around 8-16 cores (and double the threads
| number).
| 
| My naive idea(s) for fixing this issue is (I'm assuming no Haskell file
| imports unused imports here, but perhaps this may be also investigated):
| 
| 1) provide explicit dependencies which guides make to build in more
| optimal way
| 
| 2) hack GHC's make depend to kind of compute explicit dependencies from
| (1) in an optimal way automatically
| 
| 3) someone already mentioned using shake for building ghc. I don't know
| shake but perhaps this is the right direction?
| 
| 4) hack GHC to compile needed hi file directly in its memory if hi file
| is not (yet!) available (issue how to get compiling options right here).
| Also I don't know hi file semantics yet so bear with me on this.
| 
| 
| Is there anything else which may be done to fix that issue? Is someone
| already working on some of those? (I mean those reasonable from the
| list)?
| 
| Thanks!
| Karel