How to better parallelize GHC build.

[Karel Gardas]

Folks,

first of all, I remember someone already mentioned issue with decreased 
parallelism of the GHC build recently somewhere but I cann't find it 
now. Sorry, for that since otherwise I would use this thread if it was 
on this mailing list.

Anyway, while working on SPARC NCG I'm using T2000 which provides 32 
threads/8 core UltraSPARC T1 CPU. The property of this machine is that 
it's really slow on single-threaded work. To squeeze some perf from it 
man really needs to push 32 threads of work on it. Now, it really hurts 
my nerves to see it's lazy building/running just one or two ghc 
processes. To verify the fact I've created simple script to collect 
number of ghc processes over time and putting this to graph. The result 
is in the attached picture. The graph is result of running:

gmake -j64

anyway, the average number of running ghc processes is 4.4 and the 
median value is 2. IMHO such low number not only hurts build times on 
something like CMT SPARC machine, but also on let say a cluster of ARM 
machines using NFS and also on common engineering workstations which 
provide these days (IMHO!) around 8-16 cores (and double the threads 
number).

My naive idea(s) for fixing this issue is (I'm assuming no Haskell file 
imports unused imports here, but perhaps this may be also investigated):

1) provide explicit dependencies which guides make to build in more 
optimal way

2) hack GHC's make depend to kind of compute explicit dependencies from 
(1) in an optimal way automatically

3) someone already mentioned using shake for building ghc. I don't know 
shake but perhaps this is the right direction?

4) hack GHC to compile needed hi file directly in its memory if hi file 
is not (yet!) available (issue how to get compiling options right here). 
Also I don't know hi file semantics yet so bear with me on this.


Is there anything else which may be done to fix that issue? Is someone 
already working on some of those? (I mean those reasonable from the list)?

Thanks!
Karel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ghc-build.jpg
Type: image/jpeg
Size: 58003 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20150307/56dd853c/attachment-0001.jpg>