[ghc-steering-committee] #540: parallelism semaphores, recommendation: *accept*

[Eric Seidel]

Hi Committee,

Douglas Wilson, Sam Derbyshire, and Matthew Pickering have proposed adding support for a job-server to GHC. 

The goal is to allow Cabal and Stack to make optimal use of system resources when compiling many packages. Currently, the build tools are forced to choose between 

1. parallelizing across packages, but compiling each package single-threaded
2. parallelizing within packages, but compiling packages sequentially

There is no correct choice for all build plans, so the authors propose a lightweight job-server protocol to allow Cabal and GHC to cooperatively decide on the best parallelization strategy.

The implementation is already done and the changes to GHC are supposed to be quite self-contained. At a high level GHC's participation in the protocol is thus:

1. when invoked with -jsem /path/to/job-server, acquire N job tokens from the job server
2. call `setNumCapabilities N`
3. compile as usual
4. before exiting, return the tokens to the job server

There are more details in how GHC chooses how many tokens to request, and more opportunities for optimization, e.g. returning early returning of unneeded tokens, but this is really the essence of the protocol from GHC's perspective.

---

I have two minds about this proposal. On the one hand, it seems likely to leave performance on the table compared to the alternatives discussed. But on the other hand, this proposal has already been implemented and validated by Well-Typed, and it seems like a small amount of additional complexity for GHC to adapt. (Though I'd love for someone with more knowledge of GHC internals to opine on the internal complexity.)

On balance, I think we should accept this proposal and not let the perfect be the enemy of the good.

https://github.com/ghc-proposals/ghc-proposals/pull/540

Eric

On Wed, Nov 16, 2022, at 14:21, Joachim Breitner wrote:
> Hi,
>
> Am Dienstag, dem 08.11.2022 um 13:49 +0100 schrieb Joachim Breitner:
>> Dear Committee,
>> 
>> parallelism semaphores
>> have been proposed by Douglas Wilson, Sam Derbyshire, Matthew Pickering
>> 
>> https://github.com/ghc-proposals/ghc-proposals/pull/540
>> 
>> https://github.com/sheaf/ghc-proposals/blob/jsem/proposals/0540-jsem.rst
>> 
>> I suggest Adam shepherds this proposal.
>
> JFTR: I’m reassigning this to Eric (hope that’s ok for you).
>
> Cheers,
> Joachim
> -- 
> Joachim Breitner
>   mail at joachim-breitner.de
>   http://www.joachim-breitner.de/
>
> _______________________________________________
> ghc-steering-committee mailing list
> ghc-steering-committee at haskell.org
> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee