<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en" style='--code-editor-font: var(--default-mono-font, "GitLab Mono"), JetBrains Mono, Menlo, DejaVu Sans Mono, Liberation Mono, Consolas, Ubuntu Mono, Courier New, andale mono, lucida console, monospace;'>
<head>
<meta content="text/html; charset=US-ASCII" http-equiv="Content-Type">
<title>
GitLab
</title>
<style data-premailer="ignore" type="text/css">
a { color: #1068bf; }
</style>
<style>img {
max-width: 100%; height: auto;
}
body {
font-size: 0.875rem;
}
body {
-webkit-text-shadow: rgba(255,255,255,0.01) 0 0 1px;
}
body {
font-family: var(--default-regular-font, "GitLab Sans"),-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"; font-size: inherit;
}
</style>
</head>
<body style='font-size: inherit; -webkit-text-shadow: rgba(255,255,255,0.01) 0 0 1px; font-family: var(--default-regular-font, "GitLab Sans"),-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";'>
<div class="content">
<h3 style="margin-top: 20px; margin-bottom: 10px;">
Hannes Siebenhandl pushed to branch wip/fendor/ghc-iface-sharing-avoid-reserialisation at <a href="https://gitlab.haskell.org/ghc/ghc">Glasgow Haskell Compiler / GHC</a>
</h3>
<h4 style="margin-top: 10px; margin-bottom: 10px;">
Commits:
</h4>
<ul>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/5abb60f3f7f7027c6ab79a97896df054aadd4035">5abb60f3</a></strong>
<div>
<span> by Matthew Pickering </span> <i> at 2024-04-30T15:51:23+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Add run-time configurability of .hi file compression
Introduce the flag `-fwrite-if-compression=<n>` which allows to
configure the compression level of writing .hi files.
The motivation is that some deduplication operations are too expensive
for the average use case. Hence, we introduce multiple compression
levels that have a minimal impact on performance, but still reduce the
memory residency and `.hi` file size on disk considerably.
We introduce three compression levels:
* `1`: `Normal` mode. This is the least amount of compression.
It deduplicates only `Name` and `FastString`s, and is naturally the
fastest compression mode.
* `2`: `Safe` mode. It has a noticeable impact on .hi file size and is
marginally slower than `Normal` mode. In general, it should be safe to
always use `Safe` mode.
* `3`: `Full` deduplication mode. Deduplicate as much as we can,
resulting in minimal .hi files, but at the cost of additional
compilation time.
Reading .hi files doesn't need to know the initial compression level,
and can always deserialise a `ModIface`.
This allows users to experiment with different compression levels for
packages, without recompilation of dependencies.
Note, the deduplication also has an additional side effect of reduced
memory consumption to implicit sharing of deduplicated elements.
See https://gitlab.haskell.org/ghc/ghc/-/issues/24540 for example where
that matters.
-------------------------
Metric Decrease:
MultiLayerModulesDefsGhciWithCore
T16875
T21839c
T24471
hard_hole_fits
-------------------------
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/1b91e077a624a81c3667cea1e5f003d4dba3e723">1b91e077</a></strong>
<div>
<span> by Matthew Pickering </span> <i> at 2024-04-30T15:53:05+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Introduce regression tests for `.hi` file sizes
Add regression tests to track how `-fwrite-if-compression` levels affect
the size of `.hi` files.
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/5388078182762a2912f2feaae024ec465975f912">53880781</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-30T15:53:05+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Implement TrieMap for IfaceType
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/3134b745b664185371c352174b3aadf1721928d2">3134b745</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-05-06T12:00:04+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Improve sharing of duplicated values in `ModIface`
As a `ModIface` contains often duplicated values that are not
necessarily shared, we improve sharing by serialising the `ModIface`
to an in-memory byte array. Serialisation uses deduplication tables, and
deserialisation implicitly shares duplicated values.
This helps reducing the peak memory usage while compiling in
`--make` mode. The peak memory usage is especially reduced when
generating interface files with core expressions
(`-fwrite-if-simplified-core`).
On agda, this reduces the peak memory usage:
* `2.2 GB` to `1.9 GB` for a ghci session.
On `lib:Cabal`, we report:
* `570 MB` to `500 MB` for a ghci session
* `790 MB` to `667 MB` for compiling `lib:Cabal` with ghc
The execution time is not affected.
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/be43315621dedef98129332cb6f05f50e288a8bd">be433156</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-05-06T12:00:04+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>FullBinData Commit
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/cbd7902510ffeb80e96d23afcf693b684c63fa09">cbd79025</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-05-06T12:00:04+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Avoid unneccessarily re-serialising the `ModIface`
To reduce memory usage of `ModIface`, we serialise `ModIface` to an
in-memory byte array, which implicitly shares duplicated values.
This serailised byte array can be reused to avoid work when we actually
write the `ModIface` to disk.
We introduce a new field to `ModIface` which allows us to save the byte
array, and write it to disk if the `ModIface` wasn't changed after the
initial serialisation.
This requires us to change absolute offsets, for example to jump to the
deduplication table for `Name` or `FastString` with relative offsets, as
the deduplication byte array doesn't contain header information, such as
fingerprints.
To allow us to dump the binary blob to disk, we need to replace all
absolute offsets with relative ones.
This leads to new primitives for `ModIface`, which help to construct
relative offsets.
</pre>
</li>
</ul>
<h4 style="margin-top: 10px; margin-bottom: 10px;">
30 changed files:
</h4>
<ul>
<li class="file-stats">
<a href="#d0d96a6d03668aeab20ebe05e2c4ccb798c7e64c">
compiler/GHC.hs
</a>
</li>
<li class="file-stats">
<a href="#640612f2fdc707ac7ee74f4165a7b86f38aa6249">
compiler/GHC/Core/Map/Expr.hs
</a>
</li>
<li class="file-stats">
<a href="#87e780e9b5d14cf4e8a52bb125fc352fa8a1d4be">
compiler/GHC/Data/TrieMap.hs
</a>
</li>
<li class="file-stats">
<a href="#e65e7bbfc24f8830a20082bfc50f6d16e613cbeb">
compiler/GHC/Driver/DynFlags.hs
</a>
</li>
<li class="file-stats">
<a href="#9a679a2680ef6061397f1987091ea9f96ffe095d">
compiler/GHC/Driver/Main.hs
</a>
</li>
<li class="file-stats">
<a href="#774d88050336ef660c7a219fb06c480c2fc639bc">
compiler/GHC/Driver/Session.hs
</a>
</li>
<li class="file-stats">
<a href="#9f11c1c113f11a416be9e556904531050cbb5b2e">
compiler/GHC/Iface/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#699117ec0ed92a01cc9116db16477e81cd1e2ff6">
compiler/GHC/Iface/Ext/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#5df260b9f4dc378b5ad210bfabe8ca829890902c">
compiler/GHC/Iface/Ext/Fields.hs
</a>
</li>
<li class="file-stats">
<a href="#7d51edda4fa76a202cce0bca672ef300249ba8ea">
compiler/GHC/Iface/Load.hs
</a>
</li>
<li class="file-stats">
<a href="#4ed0dc87b8bdd9324e8a181cff15caa104b9e58c">
compiler/GHC/Iface/Make.hs
</a>
</li>
<li class="file-stats">
<a href="#a8e6b17fc507589522323d8dd20d9713e8c06760">
compiler/GHC/Iface/Recomp.hs
</a>
</li>
<li class="file-stats">
<a href="#d11de0586c9bcbcf4c5d10d94801cc1f97e8df1b">
compiler/GHC/Iface/Rename.hs
</a>
</li>
<li class="file-stats">
<a href="#f2b2b84d138b362920ab9f144f58bf16254aab64">
compiler/GHC/Iface/Type.hs
</a>
</li>
<li class="file-stats">
<a href="#9443aa80e92d7987999ec9c08e66c6549bdb5642">
<span class="new-file">
+
compiler/GHC/Iface/Type/Map.hs
</span>
</a>
</li>
<li class="file-stats">
<a href="#90a56cf7f130d66194a6a61cb5f372eb339ab111">
compiler/GHC/Stg/CSE.hs
</a>
</li>
<li class="file-stats">
<a href="#897218b3e24728ae9e5b0329b949493f1ede80fb">
compiler/GHC/Tc/Errors/Hole.hs
</a>
</li>
<li class="file-stats">
<a href="#9ea832ccdb4b5f7508a1efd6caedeafa16b269e7">
compiler/GHC/Tc/Gen/Splice.hs
</a>
</li>
<li class="file-stats">
<a href="#5d7c5ef0541bd4c1c950ecc95edf95171d20f538">
compiler/GHC/Tc/Utils/Backpack.hs
</a>
</li>
<li class="file-stats">
<a href="#1b4da42c16b46d6c6cc2c2962072b1e91f519f4a">
compiler/GHC/Unit/Module/ModIface.hs
</a>
</li>
<li class="file-stats">
<a href="#f35faa85de2e0a75cdaa452eb7336a09c677d450">
compiler/GHC/Utils/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#0f33fd88c617246c7f89c4477d2d1d24f942df23">
compiler/ghc.cabal.in
</a>
</li>
<li class="file-stats">
<a href="#2e45f2cd303dd998950a7c8042e796a5b2abd64f">
docs/users_guide/using-optimisation.rst
</a>
</li>
<li class="file-stats">
<a href="#0e9687963e12c5aa5ae0683ba82c078469b0753a">
<span class="new-file">
+
testsuite/tests/iface/IfaceSharingIfaceType.hs
</span>
</a>
</li>
<li class="file-stats">
<a href="#958e65b9197152a37374165a254e11c5cdc189d7">
<span class="new-file">
+
testsuite/tests/iface/IfaceSharingName.hs
</span>
</a>
</li>
<li class="file-stats">
<a href="#d73eaa55770d998038e720fe7dda2895b2849e26">
<span class="new-file">
+
testsuite/tests/iface/Lib.hs
</span>
</a>
</li>
<li class="file-stats">
<a href="#2754e39f070dc1fa5593cb16349fdffc00b579ce">
<span class="new-file">
+
testsuite/tests/iface/Makefile
</span>
</a>
</li>
<li class="file-stats">
<a href="#55f2640b716abdcd3012c5c1802b6e4be4cbfddc">
<span class="new-file">
+
testsuite/tests/iface/all.T
</span>
</a>
</li>
<li class="file-stats">
<a href="#dd22187f1b5a4b93ce5554e68adae9c811c9d395">
<span class="new-file">
+
testsuite/tests/iface/if_faststring.hs
</span>
</a>
</li>
<li class="file-stats">
<a href="#44f88eb8d33ab8c124eb468121fde2b31fb955ea">
<span class="new-file">
+
testsuite/tests/iface/if_ifacetype.hs
</span>
</a>
</li>
</ul>
<h5 style="margin-top: 10px; margin-bottom: 10px; font-size: 0.875rem;">
The diff was not included because it is too large.
</h5>
</div>
<div class="footer" style="margin-top: 10px;">
<p style="font-size: small; color: #737278;">
—
<br>
<a href="https://gitlab.haskell.org/ghc/ghc/-/compare/4bcd2b69c92d8ea2cb465a0d3188a669abdfefde...cbd7902510ffeb80e96d23afcf693b684c63fa09">View it on GitLab</a>.
<br>
You're receiving this email because of your account on <a target="_blank" rel="noopener noreferrer" href="https://gitlab.haskell.org">gitlab.haskell.org</a>. <a href="https://gitlab.haskell.org/-/profile/notifications" target="_blank" rel="noopener noreferrer" class="mng-notif-link">Manage all notifications</a> · <a href="https://gitlab.haskell.org/help" target="_blank" rel="noopener noreferrer" class="help-link">Help</a>
</p>
</div>
</body>
</html>