[Git][ghc/ghc][wip/romes/hardwire-ghc-unit-id] 23 commits: Implement -jsem: parallelism controlled by semaphores
Rodrigo Mesquita (@alt-romes)
gitlab at gitlab.haskell.org
Wed Apr 26 14:56:14 UTC 2023
Rodrigo Mesquita pushed to branch wip/romes/hardwire-ghc-unit-id at Glasgow Haskell Compiler / GHC
Commits:
5c873124 by sheaf at 2023-04-20T18:33:34-04:00
Implement -jsem: parallelism controlled by semaphores
See https://github.com/ghc-proposals/ghc-proposals/pull/540/ for a
complete description for the motivation for this feature.
The `-jsem` option allows a build tool to pass a semaphore to GHC which
GHC can use in order to control how much parallelism it requests.
GHC itself acts as a client in the GHC jobserver protocol.
```
GHC Jobserver Protocol
~~~~~~~~~~~~~~~~~~~~~~
This proposal introduces the GHC Jobserver Protocol. This protocol allows
a server to dynamically invoke many instances of a client process,
while restricting all of those instances to use no more than <n> capabilities.
This is achieved by coordination over a system semaphore (either a POSIX
semaphore [6]_ in the case of Linux and Darwin, or a Win32 semaphore [7]_
in the case of Windows platforms).
There are two kinds of participants in the GHC Jobserver protocol:
- The *jobserver* creates a system semaphore with a certain number of
available tokens.
Each time the jobserver wants to spawn a new jobclient subprocess, it **must**
first acquire a single token from the semaphore, before spawning
the subprocess. This token **must** be released once the subprocess terminates.
Once work is finished, the jobserver **must** destroy the semaphore it created.
- A *jobclient* is a subprocess spawned by the jobserver or another jobclient.
Each jobclient starts with one available token (its *implicit token*,
which was acquired by the parent which spawned it), and can request more
tokens through the Jobserver Protocol by waiting on the semaphore.
Each time a jobclient wants to spawn a new jobclient subprocess, it **must**
pass on a single token to the child jobclient. This token can either be the
jobclient's implicit token, or another token which the jobclient acquired
from the semaphore.
Each jobclient **must** release exactly as many tokens as it has acquired from
the semaphore (this does not include the implicit tokens).
```
Build tools such as cabal act as jobservers in the protocol and are
responsibile for correctly creating, cleaning up and managing the
semaphore.
Adds a new submodule (semaphore-compat) for managing and interacting
with semaphores in a cross-platform way.
Fixes #19349
- - - - -
52d3e9b4 by Ben Gamari at 2023-04-20T18:34:11-04:00
rts: Initialize Array# header in listThreads#
Previously the implementation of listThreads# failed to initialize the
header of the created array, leading to various nastiness.
Fixes #23071
- - - - -
1db30fe1 by Ben Gamari at 2023-04-20T18:34:11-04:00
testsuite: Add test for #23071
- - - - -
dae514f9 by tocic at 2023-04-21T13:31:21-04:00
Fix doc typos in libraries/base/GHC
- - - - -
113e21d7 by Sylvain Henry at 2023-04-21T13:32:01-04:00
Testsuite: replace some js_broken/js_skip predicates with req_c
Using req_c is more precise.
- - - - -
038bb031 by Krzysztof Gogolewski at 2023-04-21T18:03:04-04:00
Minor doc fixes
- Add docs/index.html to .gitignore.
It is created by ./hadrian/build docs, and it was the only file
in Hadrian's templateRules not present in .gitignore.
- Mention that MultiWayIf supports non-boolean guards
- Remove documentation of optdll - removed in 2007, 763daed95
- Fix markdown syntax
- - - - -
e826cdb2 by amesgen at 2023-04-21T18:03:44-04:00
User's guide: DeepSubsumption is implied by Haskell{98,2010}
- - - - -
499a1c20 by PHO at 2023-04-23T13:39:32-04:00
Implement executablePath for Solaris and make getBaseDir less platform-dependent
Use base-4.17 executablePath when possible, and fall back on
getExecutablePath when it's not available. The sole reason why getBaseDir
had #ifdef's was apparently that getExecutablePath wasn't reliable, and we
could reduce the number of CPP conditionals by making use of
executablePath instead.
Also export executablePath on js_HOST_ARCH.
- - - - -
97a6f7bc by tocic at 2023-04-23T13:40:08-04:00
Fix doc typos in libraries/base
- - - - -
787c6e8c by Ben Gamari at 2023-04-24T12:19:06-04:00
testsuite/T20137: Avoid impl.-defined behavior
Previously we would cast pointers to uint64_t. However, implementations
are allowed to either zero- or sign-extend such casts. Instead cast to
uintptr_t to avoid this.
Fixes #23247.
- - - - -
87095f6a by Cheng Shao at 2023-04-24T12:19:44-04:00
rts: always build 64-bit atomic ops
This patch does a few things:
- Always build 64-bit atomic ops in rts/ghc-prim, even on 32-bit
platforms
- Remove legacy "64bit" cabal flag of rts package
- Fix hs_xchg64 function prototype for 32-bit platforms
- Fix AtomicFetch test for wasm32
- - - - -
2685a12d by Cheng Shao at 2023-04-24T12:20:21-04:00
compiler: don't install signal handlers when the host platform doesn't have signals
Previously, large parts of GHC API will transitively invoke
withSignalHandlers, which doesn't work on host platforms without
signal functionality at all (e.g. wasm32-wasi). By making
withSignalHandlers a no-op on those platforms, we can make more parts
of GHC API work out of the box when signals aren't supported.
- - - - -
1338b7a3 by Cheng Shao at 2023-04-24T16:21:30-04:00
hadrian: fix non-ghc program paths passed to testsuite driver when testing cross GHC
- - - - -
1a10f556 by Bodigrim at 2023-04-24T16:22:09-04:00
Add since pragma to Data.Functor.unzip
- - - - -
0da9e882 by Soham Chowdhury at 2023-04-25T00:15:22-04:00
More informative errors for bad imports (#21826)
- - - - -
ebd5b078 by Josh Meredith at 2023-04-25T00:15:58-04:00
JS/base: provide implementation for mkdir (issue 22374)
- - - - -
8f656188 by Josh Meredith at 2023-04-25T18:12:38-04:00
JS: Fix h$base_access implementation (issue 22576)
- - - - -
74c55712 by Andrei Borzenkov at 2023-04-25T18:13:19-04:00
Give more guarntees about ImplicitParams (#23289)
- Added new section in the GHC user's guide that legends behavior of
nested implicit parameter bindings in these two cases:
let ?f = 1 in let ?f = 2 in ?f
and
data T where MkT :: (?f :: Int) => T
f :: T -> T -> Int
f MkT MkT = ?f
- Added new test case to examine this behavior.
- - - - -
c7c897a0 by Matthew Pickering at 2023-04-26T15:56:06+01:00
hadrian: Flavour: Change args -> extraArgs
Previously in a flavour definition you could override all the flags
which were passed to GHC. This causes issues when needed to compute a
package hash because we need to know what these extra arguments are
going to be before computing the hash. The solution is to modify flavour
so that the arguments you pass here are just extra ones rather than all
the arguments that you need to compile something.
This makes things work more like how cabal.project files work when you
give extra arguments to a package and also means that flavour
transformers correctly affect the hash.
- - - - -
1ad1118c by romes at 2023-04-26T15:56:06+01:00
Hardwire a better unit-id for ghc
Previously, the unit-id of ghc-the-library was fixed as `ghc`.
This was done primarily because the compiler must know the unit-id of
some packages (including ghc) a-priori to define wired-in names.
However, as seen in #20742, a reinstallable `ghc` whose unit-id is fixed
to `ghc` might result in subtle bugs when different ghc's interact.
A good example of this is having GHC_A load a plugin compiled by GHC_B,
where GHC_A and GHC_B are linked to ghc-libraries that are ABI
incompatible. Without a distinction between the unit-id of the ghc library
GHC_A is linked against and the ghc library the plugin it is loading was
compiled against, we can't check compatibility.
This patch gives a slightly better unit-id to ghc (ghc-version) by
(1) Not setting -this-unit-id to ghc, but rather to the new unit-id (modulo stage0)
(2) Adding a definition to `GHC.Settings.Config` whose value is the new unit-id.
(2.1) `GHC.Settings.Config` is generated by Hadrian
(2.2) and also by cabal through `compiler/Setup.hs`
This unit-id definition is imported by `GHC.Unit.Types` and used to
set the wired-in unit-id of "ghc", which was previously fixed to "ghc"
The commits following this one will improve the unit-id with a
cabal-style package hash and check compatibility when loading plugins.
Note that we also ensure that ghc's unit key matches unit id both when
hadrian or cabal builds ghc, and in this way we no longer need to add
`ghc` to the WiringMap.
- - - - -
032db057 by romes at 2023-04-26T15:56:06+01:00
Validate compatibility of ghcs when loading plugins
Ensure, when loading plugins, that the ghc the plugin depends on is the
ghc loading the plugin -- otherwise fail to load the plugin.
Progress towards #20742.
- - - - -
e7b682ef by romes at 2023-04-26T15:56:06+01:00
Add hashes to unit-ids created by hadrian
This commit adds support for computing an inputs hash for packages
compiled by hadrian. The result is that ABI incompatible packages should
be given different hashes and therefore be distinct in a cabal store.
Hashing is enabled by the `--flag`, and is off by default as the hash
contains a hash of the source files. We enable it when we produce
release builds so that the artifacts we distribute have the right unit
ids.
- - - - -
1e794a65 by Matthew Pickering at 2023-04-26T15:56:06+01:00
Use hash-unit-ids in release jobs
Includes fix upload_ghc_libs glob
- - - - -
30 changed files:
- .gitignore
- .gitlab/gen_ci.hs
- .gitlab/jobs.yaml
- .gitlab/rel_eng/upload_ghc_libs.py
- .gitmodules
- cabal.project-reinstall
- compiler/GHC/Driver/Make.hs
- + compiler/GHC/Driver/MakeSem.hs
- compiler/GHC/Driver/Pipeline/LogQueue.hs
- compiler/GHC/Driver/Session.hs
- compiler/GHC/Rename/Names.hs
- compiler/GHC/Rename/Unbound.hs
- compiler/GHC/Rename/Utils.hs
- compiler/GHC/Runtime/Loader.hs
- compiler/GHC/Tc/Errors/Ppr.hs
- compiler/GHC/Tc/Errors/Types.hs
- compiler/GHC/Tc/Utils/TcType.hs
- compiler/GHC/Types/Error/Codes.hs
- compiler/GHC/Types/Hint.hs
- compiler/GHC/Types/Hint/Ppr.hs
- compiler/GHC/Unit/Types.hs
- compiler/GHC/Utils/Panic.hs
- compiler/Setup.hs
- compiler/ghc.cabal.in
- configure.ac
- docs/users_guide/9.8.1-notes.rst
- docs/users_guide/exts/control.rst
- docs/users_guide/exts/implicit_parameters.rst
- docs/users_guide/exts/multiway_if.rst
- docs/users_guide/javascript.rst
The diff was not included because it is too large.
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/d4bbc74b9f211d24428a06246a88fd88813f01b0...1e794a65877eee0df6ee9aa7eb04bfbc0a5af67e
--
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/d4bbc74b9f211d24428a06246a88fd88813f01b0...1e794a65877eee0df6ee9aa7eb04bfbc0a5af67e
You're receiving this email because of your account on gitlab.haskell.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-commits/attachments/20230426/b56ac1e7/attachment-0001.html>
More information about the ghc-commits
mailing list