<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html lang="en" style='--code-editor-font: var(--default-mono-font, "GitLab Mono"), JetBrains Mono, Menlo, DejaVu Sans Mono, Liberation Mono, Consolas, Ubuntu Mono, Courier New, andale mono, lucida console, monospace;'>
<head>
<meta content="text/html; charset=US-ASCII" http-equiv="Content-Type">
<title>
GitLab
</title>
<style data-premailer="ignore" type="text/css">
a { color: #1068bf; }
</style>
<style>img {
max-width: 100%; height: auto;
}
body {
font-size: 0.875rem;
}
body {
-webkit-text-shadow: rgba(255,255,255,0.01) 0 0 1px;
}
body {
font-family: var(--default-regular-font, "GitLab Sans"),-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji"; font-size: inherit;
}
</style>
</head>
<body style='font-size: inherit; -webkit-text-shadow: rgba(255,255,255,0.01) 0 0 1px; font-family: var(--default-regular-font, "GitLab Sans"),-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";'>
<div class="content">
<h3 style="margin-top: 20px; margin-bottom: 10px;">
Hannes Siebenhandl pushed to branch wip/fendor/ifacetype-deduplication at <a href="https://gitlab.haskell.org/ghc/ghc">Glasgow Haskell Compiler / GHC</a>
</h3>
<h4 style="margin-top: 10px; margin-bottom: 10px;">
Commits:
</h4>
<ul>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/eee8f0d61a39cc5aed60c9696ccec1219113b8f2">eee8f0d6</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-22T12:13:56+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Refactor the Binary serialisation interface
The goal is simplifiy adding deduplication tables to `ModIface`
interface serialisation.
We identify two main points of interest that make this difficult:
1. UserData hardcodes what `Binary` instances can have deduplication
tables. Moreover, it heavily uses partial functions.
2. GHC.Iface.Binary hardcodes the deduplication tables for 'Name' and
'FastString', making it difficult to add more deduplication.
Instead of having a single `UserData` record with fields for all the
types that can have deduplication tables, we allow to provide custom
serialisers for any `Typeable`.
These are wrapped in existentials and stored in a `Map` indexed by their
respective `TypeRep`.
The `Binary` instance of the type to deduplicate still needs to
explicitly look up the decoder via `findUserDataReader` and
`findUserDataWriter`, which is no worse than the status-quo.
`Map` was chosen as microbenchmarks indicate it is the fastest for a
small number of keys (< 10).
To generalise the deduplication table serialisation mechanism, we
introduce the types `ReaderTable` and `WriterTable` which provide a
simple interface that is sufficient to implement a general purpose
deduplication mechanism for `writeBinIface` and `readBinIface`.
This allows us to provide a list of deduplication tables for
serialisation that can be extended more easily, for example for
`IfaceTyCon`, see the issue https://gitlab.haskell.org/ghc/ghc/-/issues/24540
for more motivation.
In addition to this refactoring, we split `UserData` into `ReaderUserData`
and `WriterUserData`, to avoid partial functions and reduce overall
memory usage, as we need fewer mutable variables.
Bump haddock submodule to accomodate for `UserData` split.
-------------------------
Metric Increase:
MultiLayerModulesTH_Make
MultiLayerModulesRecomp
T21839c
-------------------------
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/eda3aa6ee1c86bab4c07b81ce758fc4520e72f69">eda3aa6e</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-22T12:14:44+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Split `BinHandle` into `ReadBinHandle` and `WriteBinHandle`
A `BinHandle` contains too much information for reading data.
For example, it needs to keep a `FastMutInt` and a `IORef BinData`,
when the non-mutable variants would suffice.
Additionally, this change has the benefit that anyone can immediately
tell whether the `BinHandle` is used for reading or writing.
Bump haddock submodule BinHandle split.
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/303b95e91db40968dcaf1c9f6b769287ce0d82f3">303b95e9</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-22T12:19:48+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Add Eq and Ord instance to `IfaceType`
We add an `Ord` instance so that we can store `IfaceType` in a
`Data.Map` container.
This is required to deduplicate `IfaceType` while writing `.hi` files to
disk. Deduplication has many beneficial consequences to both file size
and memory usage, as the deduplication enables implicit sharing of
values.
See issue #24540 for more motivation.
The `Ord` instance would be unnecessary if we used a `TrieMap` instead
of `Data.Map` for the deduplication process. While in theory this is
clerarly the better option, experiments on the agda code base showed
that a `TrieMap` implementation has worse run-time performance
characteristics.
To the change itself, we mostly derive `Eq` and `Ord`. This requires us
to change occurrences of `FastString` with `LexicalFastString`, since
`FastString` has no `Ord` instance.
We change the definition of `IfLclName` to a newtype of
`LexicalFastString`, to make such changes in the future easier.
Bump haddock submodule for IfLclName changes
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/3a082e6efb935a8f7bca5d85a70a257b504a06fc">3a082e6e</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-22T12:19:50+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Break cyclic module dependency
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/7532c18ef2b0fea39c9bd35f3cc31aa961aa47ae">7532c18e</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-22T12:20:38+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Add deduplication table for `IfaceType`
The type `IfaceType` is a highly redundant, tree-like data structure.
While benchmarking, we realised that the high redundancy of `IfaceType`
causes high memory consumption in GHCi sessions when byte code is
embedded into the `.hi` file via `-fwrite-if-simplified-core` or
`-fbyte-code-and-object-code`.
Loading such `.hi` files from disk introduces many duplicates of
memory expensive values in `IfaceType`, such as `IfaceTyCon`,
`IfaceTyConApp`, `IA_Arg` and many more.
We improve the memory behaviour of GHCi by adding an additional
deduplication table for `IfaceType` to the serialisation of `ModIface`,
similar to how we deduplicate `Name`s and `FastString`s.
When reading the interface file back, the table allows us to automatically
share identical values of `IfaceType`.
To provide some numbers, we evaluated this patch on the agda code base.
We loaded the full library from the `.hi` files, which contained the
embedded core expressions (`-fwrite-if-simplified-core`).
Before this patch:
* Load time: 11.7 s, 2.5 GB maximum residency.
After this patch:
* Load time: 7.3 s, 1.7 GB maximum residency.
This deduplication has the beneficial side effect to additionally reduce
the size of the on-disk interface files tremendously.
For example, on agda, we reduce the size of `.hi` files (with
`-fwrite-if-simplified-core`):
* Before: 101 MB on disk
* Now: 24 MB on disk
This has even a beneficial side effect on the cabal store. We reduce the
size of the store on disk:
* Before: 341 MB on disk
* Now: 310 MB on disk
Note, none of the dependencies have been compiled with
`-fwrite-if-simplified-core`, but `IfaceType` occurs in multiple
locations in a `ModIface`.
We also add IfaceType deduplication table to .hie serialisation and
refactor .hie file serialisation to use the same infrastrucutre as
`putWithTables`.
Bump haddock submodule to accomodate for changes to the deduplication
table layout and binary interface.
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/b3dbf7bd4d17f7057e960277c6e8afda6e25013c">b3dbf7bd</a></strong>
<div>
<span> by Matthew Pickering </span> <i> at 2024-04-22T12:21:21+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Add run-time configurability of .hi file compression
Introduce the flag `-fwrite-if-compression=<n>` which allows to
configure the compression level of writing .hi files.
The motivation is that some deduplication operations are too expensive
for the average use case. Hence, we introduce multiple compression
levels that have a minimal impact on performance, but still reduce the
memory residency and `.hi` file size on disk considerably.
We introduce three compression levels:
* `1`: `Normal` mode. This is the least amount of compression.
It deduplicates only `Name` and `FastString`s, and is naturally the
fastest compression mode.
* `2`: `Safe` mode. It has a noticeable impact on .hi file size and is
marginally slower than `Normal` mode. In general, it should be safe to
always use `Safe` mode.
* `3`: `Full` deduplication mode. Deduplicate as much as we can,
resulting in minimal .hi files, but at the cost of additional
compilation time.
Reading .hi files doesn't need to know the initial compression level,
and can always deserialise a `ModIface`.
This allows users to experiment with different compression levels for
packages, without recompilation of dependencies.
Note, the deduplication also has an additional side effect of reduced
memory consumption to implicit sharing of deduplicated elements.
See https://gitlab.haskell.org/ghc/ghc/-/issues/24540 for example where
that matters.
-------------------------
Metric Decrease:
T21839c
T24471
-------------------------
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/70dd312b5449aa896b181398f53d1d813eec6697">70dd312b</a></strong>
<div>
<span> by Matthew Pickering </span> <i> at 2024-04-22T12:21:21+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Introduce regression tests for `.hi` file sizes
Add regression tests to track how `-fwrite-if-compression` levels affect
the size of `.hi` files.
</pre>
</li>
<li>
<strong style="font-weight: bold;"><a href="https://gitlab.haskell.org/ghc/ghc/-/commit/2fc1995a1704d41a3bea6cac7edd172979a5c5e1">2fc1995a</a></strong>
<div>
<span> by Fendor </span> <i> at 2024-04-22T12:21:21+02:00 </i>
</div>
<pre class="commit-message" style='white-space: pre-wrap; display: block; font-size: 14px; color: #333238; position: relative; font-family: var(--default-mono-font, "GitLab Mono"),"JetBrains Mono","Menlo","DejaVu Sans Mono","Liberation Mono","Consolas","Ubuntu Mono","Courier New","andale mono","lucida console",monospace; word-break: break-all; word-wrap: break-word; background-color: #fbfafd; border-radius: 2px; margin: 0; padding: 8px 12px; border: 1px solid #dcdcde;'>Implement TrieMap for IfaceType
</pre>
</li>
</ul>
<h4 style="margin-top: 10px; margin-bottom: 10px;">
30 changed files:
</h4>
<ul>
<li class="file-stats">
<a href="#640612f2fdc707ac7ee74f4165a7b86f38aa6249">
compiler/GHC/Core/Map/Expr.hs
</a>
</li>
<li class="file-stats">
<a href="#578c7c3857d66d963736ff6742f3433a0e8d01b7">
compiler/GHC/Core/TyCo/Rep.hs
</a>
</li>
<li class="file-stats">
<a href="#bbf06eb5d4f0db7bc3213b84b9f5822710ff7be8">
compiler/GHC/CoreToIface.hs
</a>
</li>
<li class="file-stats">
<a href="#d51029aec6ce58e66d4b48865c1dd0d2038fda29">
compiler/GHC/Data/FastString.hs
</a>
</li>
<li class="file-stats">
<a href="#87e780e9b5d14cf4e8a52bb125fc352fa8a1d4be">
compiler/GHC/Data/TrieMap.hs
</a>
</li>
<li class="file-stats">
<a href="#e65e7bbfc24f8830a20082bfc50f6d16e613cbeb">
compiler/GHC/Driver/DynFlags.hs
</a>
</li>
<li class="file-stats">
<a href="#9a679a2680ef6061397f1987091ea9f96ffe095d">
compiler/GHC/Driver/Main.hs
</a>
</li>
<li class="file-stats">
<a href="#774d88050336ef660c7a219fb06c480c2fc639bc">
compiler/GHC/Driver/Session.hs
</a>
</li>
<li class="file-stats">
<a href="#9f11c1c113f11a416be9e556904531050cbb5b2e">
compiler/GHC/Iface/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#5fbeca6b3e6a959dedea83c4af51c13942e2def5">
compiler/GHC/Iface/Decl.hs
</a>
</li>
<li class="file-stats">
<a href="#12fae53271ab8929046115ab04711a1075fef5bf">
compiler/GHC/Iface/Env.hs
</a>
</li>
<li class="file-stats">
<a href="#699117ec0ed92a01cc9116db16477e81cd1e2ff6">
compiler/GHC/Iface/Ext/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#5df260b9f4dc378b5ad210bfabe8ca829890902c">
compiler/GHC/Iface/Ext/Fields.hs
</a>
</li>
<li class="file-stats">
<a href="#1cf39038e57acc2a80cb5fb3fef9733e8e37b8fb">
compiler/GHC/Iface/Ext/Utils.hs
</a>
</li>
<li class="file-stats">
<a href="#7d51edda4fa76a202cce0bca672ef300249ba8ea">
compiler/GHC/Iface/Load.hs
</a>
</li>
<li class="file-stats">
<a href="#a8e6b17fc507589522323d8dd20d9713e8c06760">
compiler/GHC/Iface/Recomp.hs
</a>
</li>
<li class="file-stats">
<a href="#19bf6e34aab7827cc9e5b0eea75f2380efb11735">
compiler/GHC/Iface/Recomp/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#0b4bc77723e1f22f50d70e89a81f6d92c3904408">
compiler/GHC/Iface/Recomp/Flags.hs
</a>
</li>
<li class="file-stats">
<a href="#079d6b28856775716c78fdd4a4b35bb45a59002c">
compiler/GHC/Iface/Syntax.hs
</a>
</li>
<li class="file-stats">
<a href="#f2b2b84d138b362920ab9f144f58bf16254aab64">
compiler/GHC/Iface/Type.hs
</a>
</li>
<li class="file-stats">
<a href="#9443aa80e92d7987999ec9c08e66c6549bdb5642">
<span class="new-file">
+
compiler/GHC/Iface/Type/Map.hs
</span>
</a>
</li>
<li class="file-stats">
<a href="#06ff7bac58fd8cfe0c17b81963b03f4cce86a065">
compiler/GHC/IfaceToCore.hs
</a>
</li>
<li class="file-stats">
<a href="#90a56cf7f130d66194a6a61cb5f372eb339ab111">
compiler/GHC/Stg/CSE.hs
</a>
</li>
<li class="file-stats">
<a href="#f62b0cd33586bc34bf5ea863bd9748f7f3e38b77">
compiler/GHC/StgToJS/Object.hs
</a>
</li>
<li class="file-stats">
<a href="#166e3514d27c63ef7e86af29830d9e8b2a31c8b1">
compiler/GHC/Types/Basic.hs
</a>
</li>
<li class="file-stats">
<a href="#e995173a19cd0878fdecbafad6e371fc293fd878">
compiler/GHC/Types/FieldLabel.hs
</a>
</li>
<li class="file-stats">
<a href="#8905161187cfddae9e9ece36910196e521651b00">
compiler/GHC/Types/Name.hs
</a>
</li>
<li class="file-stats">
<a href="#5dac619989e8509003cf7b463c100f5cc56072a3">
compiler/GHC/Types/Var.hs
</a>
</li>
<li class="file-stats">
<a href="#f35faa85de2e0a75cdaa452eb7336a09c677d450">
compiler/GHC/Utils/Binary.hs
</a>
</li>
<li class="file-stats">
<a href="#3b51e7209da44ef16b6e67f66c5bc374891fe587">
compiler/GHC/Utils/Binary/Typeable.hs
</a>
</li>
</ul>
<h5 style="margin-top: 10px; margin-bottom: 10px; font-size: 0.875rem;">
The diff was not included because it is too large.
</h5>
</div>
<div class="footer" style="margin-top: 10px;">
<p style="font-size: small; color: #737278;">
—
<br>
<a href="https://gitlab.haskell.org/ghc/ghc/-/compare/c79e84bef5b9c45eaaa21de9d4fcc19bdcaeb4f5...2fc1995a1704d41a3bea6cac7edd172979a5c5e1">View it on GitLab</a>.
<br>
You're receiving this email because of your account on <a target="_blank" rel="noopener noreferrer" href="https://gitlab.haskell.org">gitlab.haskell.org</a>. <a href="https://gitlab.haskell.org/-/profile/notifications" target="_blank" rel="noopener noreferrer" class="mng-notif-link">Manage all notifications</a> · <a href="https://gitlab.haskell.org/help" target="_blank" rel="noopener noreferrer" class="help-link">Help</a>
</p>
</div>
</body>
</html>