Parallel --make (GHC build times on newer MacBook Pros?)

[Simon Marlow]

On 03/09/2011 02:05, Evan Laforge wrote:
>>> Another way to do this would be to have GHC --make invoke itself to
>>> compile each module separately.  Actually I think I prefer this method,
>>> although it might be a bit slower since each individual compilation has
>>> to read lots of interface files.  The main GHC --make process would do
>>> the final link only.  A fun hack for somebody?
>>
>> this would also help building large libraries on architectures with
>> little memory, as it seems to me that when one ghc instance is compiling
>> multiple modules in a row, some leaked memory/unevaluated thunks pile up
>> and eventually cause the compilation to abort. I suspect that building
>> each file on its own avoids this issue.
>
> In my experience, reading all those .hi files is not so quick, about
> 1.5s for around 200 modules, on an SSD.  It gets worse with a pgmF, since ghc
> wants to preprocess each file, it's a minimum of 5s given 'cat' as a
> preprocessor.
>
> Part of my wanting to use make instead of --make was to avoid this
> re-preprocessing delay.  It's nice that it will automatically notice
> which modules to recompile if a CPP define changes, but not so nice
> that it has to take a lot of time to figure that out every single
> compile, or for a preprocessor that doesn't have the power to change
> whether the module should be recompiled or not.

Ah, but you're measuring the startup time of ghc --make, which is not 
the same as the work that each individual ghc would do if ghc were 
invoked separately on each module, for two reasons:

  - when used in one-shot mode (i.e. without --make), ghc only reads
    and processes the interface files it needs, lazilly

  - the individual ghc's would not need to proprocess modules - that
    would only be done once, by the master process, before starting
    the subprocesses.  The preprocessed source would be cached,
    exactly as it is now by --make.

Cheers,
	Simon