[ghc-steering-committee] #540: parallelism semaphores, recommendation: *accept*
Eric Seidel
eric at seidel.io
Thu Feb 23 18:07:32 UTC 2023
Hi all,
We have votes in favor from Joachim, Arnaud, Chris, Moritz, and Simon M, and silence otherwise.
Simon PJ, Richard, Vlad, Adam: if you have thoughts or concerns about this proposal, please speak up!
On Sun, Feb 12, 2023, at 12:01, Eric Seidel wrote:
> Hi Committee,
>
> Douglas Wilson, Sam Derbyshire, and Matthew Pickering have proposed
> adding support for a job-server to GHC.
>
> The goal is to allow Cabal and Stack to make optimal use of system
> resources when compiling many packages. Currently, the build tools are
> forced to choose between
>
> 1. parallelizing across packages, but compiling each package single-threaded
> 2. parallelizing within packages, but compiling packages sequentially
>
> There is no correct choice for all build plans, so the authors propose
> a lightweight job-server protocol to allow Cabal and GHC to
> cooperatively decide on the best parallelization strategy.
>
> The implementation is already done and the changes to GHC are supposed
> to be quite self-contained. At a high level GHC's participation in the
> protocol is thus:
>
> 1. when invoked with -jsem /path/to/job-server, acquire N job tokens
> from the job server
> 2. call `setNumCapabilities N`
> 3. compile as usual
> 4. before exiting, return the tokens to the job server
>
> There are more details in how GHC chooses how many tokens to request,
> and more opportunities for optimization, e.g. returning early returning
> of unneeded tokens, but this is really the essence of the protocol from
> GHC's perspective.
>
> ---
>
> I have two minds about this proposal. On the one hand, it seems likely
> to leave performance on the table compared to the alternatives
> discussed. But on the other hand, this proposal has already been
> implemented and validated by Well-Typed, and it seems like a small
> amount of additional complexity for GHC to adapt. (Though I'd love for
> someone with more knowledge of GHC internals to opine on the internal
> complexity.)
>
> On balance, I think we should accept this proposal and not let the
> perfect be the enemy of the good.
>
> https://github.com/ghc-proposals/ghc-proposals/pull/540
>
> Eric
>
> On Wed, Nov 16, 2022, at 14:21, Joachim Breitner wrote:
>> Hi,
>>
>> Am Dienstag, dem 08.11.2022 um 13:49 +0100 schrieb Joachim Breitner:
>>> Dear Committee,
>>>
>>> parallelism semaphores
>>> have been proposed by Douglas Wilson, Sam Derbyshire, Matthew Pickering
>>>
>>> https://github.com/ghc-proposals/ghc-proposals/pull/540
>>>
>>> https://github.com/sheaf/ghc-proposals/blob/jsem/proposals/0540-jsem.rst
>>>
>>> I suggest Adam shepherds this proposal.
>>
>> JFTR: I’m reassigning this to Eric (hope that’s ok for you).
>>
>> Cheers,
>> Joachim
>> --
>> Joachim Breitner
>> mail at joachim-breitner.de
>> http://www.joachim-breitner.de/
>>
>> _______________________________________________
>> ghc-steering-committee mailing list
>> ghc-steering-committee at haskell.org
>> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee
More information about the ghc-steering-committee
mailing list