[ghc-steering-committee] #540: parallelism semaphores, recommendation: *accept*

[Eric Seidel]

Hi all,

We have votes in favor from Joachim, Arnaud, Chris, Moritz, and Simon M, and silence otherwise.

Simon PJ, Richard, Vlad, Adam: if you have thoughts or concerns about this proposal, please speak up! 

On Sun, Feb 12, 2023, at 12:01, Eric Seidel wrote:
> Hi Committee,
>
> Douglas Wilson, Sam Derbyshire, and Matthew Pickering have proposed 
> adding support for a job-server to GHC. 
>
> The goal is to allow Cabal and Stack to make optimal use of system 
> resources when compiling many packages. Currently, the build tools are 
> forced to choose between 
>
> 1. parallelizing across packages, but compiling each package single-threaded
> 2. parallelizing within packages, but compiling packages sequentially
>
> There is no correct choice for all build plans, so the authors propose 
> a lightweight job-server protocol to allow Cabal and GHC to 
> cooperatively decide on the best parallelization strategy.
>
> The implementation is already done and the changes to GHC are supposed 
> to be quite self-contained. At a high level GHC's participation in the 
> protocol is thus:
>
> 1. when invoked with -jsem /path/to/job-server, acquire N job tokens 
> from the job server
> 2. call `setNumCapabilities N`
> 3. compile as usual
> 4. before exiting, return the tokens to the job server
>
> There are more details in how GHC chooses how many tokens to request, 
> and more opportunities for optimization, e.g. returning early returning 
> of unneeded tokens, but this is really the essence of the protocol from 
> GHC's perspective.
>
> ---
>
> I have two minds about this proposal. On the one hand, it seems likely 
> to leave performance on the table compared to the alternatives 
> discussed. But on the other hand, this proposal has already been 
> implemented and validated by Well-Typed, and it seems like a small 
> amount of additional complexity for GHC to adapt. (Though I'd love for 
> someone with more knowledge of GHC internals to opine on the internal 
> complexity.)
>
> On balance, I think we should accept this proposal and not let the 
> perfect be the enemy of the good.
>
> https://github.com/ghc-proposals/ghc-proposals/pull/540
>
> Eric
>
> On Wed, Nov 16, 2022, at 14:21, Joachim Breitner wrote:
>> Hi,
>>
>> Am Dienstag, dem 08.11.2022 um 13:49 +0100 schrieb Joachim Breitner:
>>> Dear Committee,
>>> 
>>> parallelism semaphores
>>> have been proposed by Douglas Wilson, Sam Derbyshire, Matthew Pickering
>>> 
>>> https://github.com/ghc-proposals/ghc-proposals/pull/540
>>> 
>>> https://github.com/sheaf/ghc-proposals/blob/jsem/proposals/0540-jsem.rst
>>> 
>>> I suggest Adam shepherds this proposal.
>>
>> JFTR: I’m reassigning this to Eric (hope that’s ok for you).
>>
>> Cheers,
>> Joachim
>> -- 
>> Joachim Breitner
>>   mail at joachim-breitner.de
>>   http://www.joachim-breitner.de/
>>
>> _______________________________________________
>> ghc-steering-committee mailing list
>> ghc-steering-committee at haskell.org
>> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee