[ghc-steering-committee] #540: parallelism semaphores, recommendation: *accept*

[Simon Marlow]

Yes this seems entirely reasonable, +1.

Simon

On Sun, 12 Feb 2023 at 19:02, Eric Seidel <eric at seidel.io> wrote:

> Hi Committee,
>
> Douglas Wilson, Sam Derbyshire, and Matthew Pickering have proposed adding
> support for a job-server to GHC.
>
> The goal is to allow Cabal and Stack to make optimal use of system
> resources when compiling many packages. Currently, the build tools are
> forced to choose between
>
> 1. parallelizing across packages, but compiling each package
> single-threaded
> 2. parallelizing within packages, but compiling packages sequentially
>
> There is no correct choice for all build plans, so the authors propose a
> lightweight job-server protocol to allow Cabal and GHC to cooperatively
> decide on the best parallelization strategy.
>
> The implementation is already done and the changes to GHC are supposed to
> be quite self-contained. At a high level GHC's participation in the
> protocol is thus:
>
> 1. when invoked with -jsem /path/to/job-server, acquire N job tokens from
> the job server
> 2. call `setNumCapabilities N`
> 3. compile as usual
> 4. before exiting, return the tokens to the job server
>
> There are more details in how GHC chooses how many tokens to request, and
> more opportunities for optimization, e.g. returning early returning of
> unneeded tokens, but this is really the essence of the protocol from GHC's
> perspective.
>
> ---
>
> I have two minds about this proposal. On the one hand, it seems likely to
> leave performance on the table compared to the alternatives discussed. But
> on the other hand, this proposal has already been implemented and validated
> by Well-Typed, and it seems like a small amount of additional complexity
> for GHC to adapt. (Though I'd love for someone with more knowledge of GHC
> internals to opine on the internal complexity.)
>
> On balance, I think we should accept this proposal and not let the perfect
> be the enemy of the good.
>
> https://github.com/ghc-proposals/ghc-proposals/pull/540
>
> Eric
>
> On Wed, Nov 16, 2022, at 14:21, Joachim Breitner wrote:
> > Hi,
> >
> > Am Dienstag, dem 08.11.2022 um 13:49 +0100 schrieb Joachim Breitner:
> >> Dear Committee,
> >>
> >> parallelism semaphores
> >> have been proposed by Douglas Wilson, Sam Derbyshire, Matthew Pickering
> >>
> >> https://github.com/ghc-proposals/ghc-proposals/pull/540
> >>
> >>
> https://github.com/sheaf/ghc-proposals/blob/jsem/proposals/0540-jsem.rst
> >>
> >> I suggest Adam shepherds this proposal.
> >
> > JFTR: I’m reassigning this to Eric (hope that’s ok for you).
> >
> > Cheers,
> > Joachim
> > --
> > Joachim Breitner
> >   mail at joachim-breitner.de
> >   http://www.joachim-breitner.de/
> >
> > _______________________________________________
> > ghc-steering-committee mailing list
> > ghc-steering-committee at haskell.org
> > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee
> _______________________________________________
> ghc-steering-committee mailing list
> ghc-steering-committee at haskell.org
> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-steering-committee
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-steering-committee/attachments/20230214/29d6783c/attachment.html>