[ghc-steering-committee] #540: parallelism semaphores, recommendation: *accept*

[Richard Eisenberg]

I didn't know we were voting on this one!

I requested a loud Haddock comment on GitHub, but I'm in support.

Richard

> On Feb 23, 2023, at 1:07 PM, Eric Seidel <eric at seidel.io> wrote:
> 
> Hi all,
> 
> We have votes in favor from Joachim, Arnaud, Chris, Moritz, and Simon M, and silence otherwise.
> 
> Simon PJ, Richard, Vlad, Adam: if you have thoughts or concerns about this proposal, please speak up! 
> 
> On Sun, Feb 12, 2023, at 12:01, Eric Seidel wrote:
>> Hi Committee,
>> 
>> Douglas Wilson, Sam Derbyshire, and Matthew Pickering have proposed 
>> adding support for a job-server to GHC. 
>> 
>> The goal is to allow Cabal and Stack to make optimal use of system 
>> resources when compiling many packages. Currently, the build tools are 
>> forced to choose between 
>> 
>> 1. parallelizing across packages, but compiling each package single-threaded
>> 2. parallelizing within packages, but compiling packages sequentially
>> 
>> There is no correct choice for all build plans, so the authors propose 
>> a lightweight job-server protocol to allow Cabal and GHC to 
>> cooperatively decide on the best parallelization strategy.
>> 
>> The implementation is already done and the changes to GHC are supposed 
>> to be quite self-contained. At a high level GHC's participation in the 
>> protocol is thus:
>> 
>> 1. when invoked with -jsem /path/to/job-server, acquire N job tokens 
>> from the job server
>> 2. call `setNumCapabilities N`
>> 3. compile as usual
>> 4. before exiting, return the tokens to the job server
>> 
>> There are more details in how GHC chooses how many tokens to request, 
>> and more opportunities for optimization, e.g. returning early returning 
>> of unneeded tokens, but this is really the essence of the protocol from 
>> GHC's perspective.
>> 
>> ---
>> 
>> I have two minds about this proposal. On the one hand, it seems likely 
>> to leave performance on the table compared to the alternatives 
>> discussed. But on the other hand, this proposal has already been 
>> implemented and validated by Well-Typed, and it seems like a small 
>> amount of additional complexity for GHC to adapt. (Though I'd love for 
>> someone with more knowledge of GHC internals to opine on the internal 
>> complexity.)
>> 
>> On balance, I think we should accept this proposal and not let the 
>> perfect be the enemy of the good.
>> 
>> https://github.com/ghc-proposals/ghc-proposals/pull/540
>> 
>> Eric
>> 
>> On Wed, Nov 16, 2022, at 14:21, Joachim Breitner wrote:
>>> Hi,
>>> 
>>> Am Dienstag, dem 08.11.2022 um 13:49 +0100 schrieb Joachim Breitner:
>>>> Dear Committee,
>>>> 
>>>> parallelism semaphores
>>>> have been proposed by Douglas Wilson, Sam Derbyshire, Matthew Pickering
>>>> 
>>>> https://github.com/ghc-proposals/ghc-proposals/pull/540
>>>> 
>>>> https://github.com/sheaf/ghc-proposals/blob/jsem/proposals/0540-jsem.rst
>>>> 
>>>> I suggest Adam shepherds this proposal.
>>> 
>>> JFTR: I’m reassigning this to Eric (hope that’s ok for you).
>>> 
>>> Cheers,
>>> Joachim
>>> -- 
>>> Joachim Breitner
>>>  mail at joachim-breitner.de
>>>  http://www.joachim-breitner.de/
>>> 
>>> _______________________________________________
>>> ghc-steering-committee mailing list
>>> ghc-steering-committee at haskell.org
>>> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee
> _______________________________________________
> ghc-steering-committee mailing list
> ghc-steering-committee at haskell.org
> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee