<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I agree one should be able to get most of the testing value from
stage1. And the tooling team at IOHK has done some work in
<a class="moz-txt-link-freetext" href="https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3652">https://gitlab.haskell.org/ghc/ghc/-/merge_requests/3652</a> to allow
a stage 1 compiler to be tested. That's a very important first
step!<br>
</p>
<p>But TH and GHCi require either iserv (external interpreter) or an
compiler whose own ABI and the outputted ABI match for the
internal interpreter, and ideally we should test both. I think
doing a --freeze1 stage2 build *in addition* to the stage1 build
would work in the majority of cases, and that would allow us to
incrementally build and test both. Remember that iserv uses the
ghc library, and needs to be ABI comparable with the stage1
compiler that is using it, so it is less a panacea than it might
seem like for ABI changes vs mere cross compilation.<br>
</p>
<p>I opened
<a class="moz-txt-link-freetext" href="https://github.com/ghc-proposals/ghc-proposals/issues/162">https://github.com/ghc-proposals/ghc-proposals/issues/162</a> for an
ABI-agnostic interpreter that would allow stage1 alone to do GHCi
and TH a third away unconditionally. This would also allow TH to
safely be used in GHC itself, but for the purposes of this
discussion, it's nice to make testing more reliable without the
--freeze1 stage 2 gamble.<br>
</p>
<p>Bottom line is, yes, building stage 2 from a freshly-built stage
1 will invalidate any cache, and so we should avoid that.<br>
</p>
<p>John<br>
</p>
<div class="moz-cite-prefix">On 2/22/21 8:42 AM, Spiwack, Arnaud
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAH6fHQninD3ZPvesVk9FUKc4b1sFjs=jG5XmtpOeD=pVsMjfQg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>Let me know if I'm talking nonsense, but I believe that we
are building both stages for each architecture and flavour. Do
we need to build two stages everywhere? What stops us from
building a single stage? And if anything, what can we change
to get into a situation where we can?</div>
<div><br>
</div>
<div>Quite better than reusing build incrementally, is not
building at all.<br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Feb 22, 2021 at 10:09
AM Simon Peyton Jones via ghc-devs <<a
href="mailto:ghc-devs@haskell.org" moz-do-not-send="true">ghc-devs@haskell.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" lang="EN-GB">
<div class="gmail-m_6346696964856726737WordSection1">
<p style="margin-left:36pt">Incremental CI can cut
multiple hours to < mere minutes, especially with the
test suite being embarrassingly parallel. There simply
no way optimizations to the compiler independent from
sharing a cache between CI runs can get anywhere close
to that return on investment.</p>
<p class="MsoNormal"><span>I rather agree with this. I
don’t think there is much low-hanging fruit on compile
times, aside from coercion-zapping which we are
working on anyway. If we got a 10% reduction in
compile time we’d be over the moon, but our users
would barely notice.</span></p>
<p class="MsoNormal"><span> </span></p>
<p class="MsoNormal"><span>To get truly substantial
improvements (a factor of 2 or 10) I think we need to
do less compiling – hence incremental CI.</span></p>
<p class="MsoNormal"><span><br>
Simon</span></p>
<p class="MsoNormal"><span> </span></p>
<div style="border-color:currentcolor currentcolor
currentcolor blue;border-style:none none none
solid;border-width:medium medium medium
1.5pt;padding:0cm 0cm 0cm 4pt">
<div>
<div style="border-color:rgb(225,225,225) currentcolor
currentcolor;border-style:solid none
none;border-width:1pt medium medium;padding:3pt 0cm
0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span
lang="EN-US"> ghc-devs <<a
href="mailto:ghc-devs-bounces@haskell.org"
target="_blank" moz-do-not-send="true">ghc-devs-bounces@haskell.org</a>>
<b>On Behalf Of </b>John Ericson<br>
<b>Sent:</b> 22 February 2021 05:53<br>
<b>To:</b> ghc-devs <<a
href="mailto:ghc-devs@haskell.org"
target="_blank" moz-do-not-send="true">ghc-devs@haskell.org</a>><br>
<b>Subject:</b> Re: On CI</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p>I'm not opposed to some effort going into this, but I
would strongly opposite putting all our effort there.
Incremental CI can cut multiple hours to < mere
minutes, especially with the test suite being
embarrassingly parallel. There simply no way
optimizations to the compiler independent from sharing
a cache between CI runs can get anywhere close to that
return on investment.</p>
<p>(FWIW, I'm also skeptical that the people complaining
about GHC performance know what's hurting them most.
For example, after non-incrementality, the next
slowest thing is linking, which is...not done by GHC!
But all that is a separate conversation.)</p>
<p>John</p>
<div>
<p class="MsoNormal">On 2/19/21 2:42 PM, Richard
Eisenberg wrote:</p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal">There are some good ideas here,
but I want to throw out another one: put all our
effort into reducing compile times. There is a loud
plea to do this on <a
href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdiscourse.haskell.org%2Ft%2Fcall-for-ideas-forming-a-technical-agenda%2F1901%2F24&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691120329%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=1CV0MEVUZpbAbmKAWTIiqLgjft7IbN%2BCSnvB3W3iX%2FU%3D&reserved=0"
target="_blank" moz-do-not-send="true">Discourse</a>,
and it would both solve these CI problems and also
help everyone else. </p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">This isn't to say to stop
exploring the ideas here. But since time is mostly
fixed, tackling compilation times in general may
be the best way out of this. Ben's survey of other
projects (thanks!) shows that we're way, way
behind in how long our CI takes to run.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Richard</p>
<div>
<p class="MsoNormal"><br>
<br>
</p>
<blockquote
style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">On Feb 19, 2021, at 7:20
AM, Sebastian Graf <<a
href="mailto:sgraf1337@gmail.com"
target="_blank" moz-do-not-send="true">sgraf1337@gmail.com</a>>
wrote:</p>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<div>
<p class="MsoNormal"><span
style="font-size:13.5pt">Recompilation
avoidance</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I think in order to
cache more in CI, we first have to
invest some time in fixing recompilation
avoidance in our bootstrapped build
system.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">I just tested on a
hadrian perf ticky build: Adding one
line of *comment* in the compiler causes</p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal">
a (pretty slow, yet negligible)
rebuild of the stage1 compiler</li>
<li class="MsoNormal">
2 minutes of RTS rebuilding (Why do we
have to rebuild the RTS? It doesn't
depend in any way on the change I
made)</li>
<li class="MsoNormal">
apparent full rebuild the libraries</li>
<li class="MsoNormal">
apparent full rebuild of the stage2
compiler</li>
</ul>
<div>
<p class="MsoNormal">That took 17
minutes, a full build takes
~45minutes. So there definitely is
some caching going on, but not nearly
as much as there could be.</p>
</div>
<div>
<p class="MsoNormal">I know there have
been great and boring efforts on
compiler determinism in the past, but
either it's not good enough or our
build system needs fixing.</p>
</div>
<div>
<p class="MsoNormal">I think a good
first step to assert would be to make
sure that the hash of the stage1
compiler executable doesn't change if
I only change a comment.</p>
</div>
<div>
<p class="MsoNormal">I'm aware there
probably is stuff going on, like
embedding configure dates in interface
files and executables, that would need
to go, but if possible this would be a
huge improvement.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">On the other hand,
we can simply tack on a [skip ci] to
the commit message, as I did for <a
href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fmerge_requests%2F4975&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691130329%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=bgT0LeZXjF%2BMklzctvZL6WaVpaddN7%2FSpojcEXGXv7Q%3D&reserved=0"
target="_blank"
moz-do-not-send="true">https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4975</a>.
Variants like [skip tests] or
[frontend] could help to identify
which tests to run by default.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:13.5pt">Lean</span></p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal">I had a chat with a
colleague about how they do CI for Lean.
Apparently, CI turnaround time including
tests is generally 25 minutes (~15
minutes for the build) for a complete
pipeline, testing 6 different OSes and
configurations in parallel:
<a
href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fleanprover%2Flean4%2Factions%2Fworkflows%2Fci.yml&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691140326%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9MEWPlRhO2xZK2iu5OqzXS9RZqc9pKNJcGDv7Nj3hyA%3D&reserved=0"
target="_blank" moz-do-not-send="true">
https://github.com/leanprover/lean4/actions/workflows/ci.yml</a></p>
</div>
<div>
<p class="MsoNormal">They utilise ccache
to cache the clang-based C++-backend, so
that they only have to re-run the front-
and middle-end. In effect, they take
advantage of the fact that the
"function" clang, in contrast to the
"function" stage1 compiler, stays the
same.</p>
</div>
<div>
<p class="MsoNormal">It's hard to achieve
that for GHC, where a complete compiler
pipeline comes as one big, fused
"function": An external tool can never
be certain that a change to Parser.y
could not affect the CodeGen phase.</p>
</div>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">Inspired by Lean, the
following is a bit inconcrete and
imaginary, but maybe we could make it so
that compiler phases "sign" parts of the
interface file with the binary hash of
the respective subcomponents of the
phase?</p>
</div>
<div>
<p class="MsoNormal">E.g., if all the
object files that influence CodeGen
(that will later be linked into the
stage1 compiler) result in a hash of
0xdeadbeef before and after the change
to Parser.y, we know we can stop
recompiling Data.List with the stage1
compiler when we see that the IR passed
to CodeGen didn't change, because the
last compile did CodeGen with a stage1
compiler with the same hash 0xdeadbeef.
The 0xdeadbeef hash is a proxy for
saying "the function CodeGen stayed the
same", so we can reuse its cached
outputs.</p>
</div>
<div>
<p class="MsoNormal">Of course, that is
utopic without a tool that does the
"taint analysis" of which modules in GHC
influence CodeGen. Probably just
including all the transitive
dependencies of GHC.CmmToAsm suffices,
but probably that's too crude already.
For another example, a change to
GHC.Utils.Unique would probably entail a
full rebuild of the compiler because it
basically affects all compiler phases.</p>
</div>
<div>
<p class="MsoNormal">There are probably
parallels with recompilation avoidance
in a language with staged
meta-programming.</p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal">Am Fr., 19. Feb. 2021
um 11:42 Uhr schrieb Josef Svenningsson
via ghc-devs <<a
href="mailto:ghc-devs@haskell.org"
target="_blank" moz-do-not-send="true">ghc-devs@haskell.org</a>>:</p>
</div>
<blockquote style="border-color:currentcolor
currentcolor currentcolor
rgb(204,204,204);border-style:none none
none solid;border-width:medium medium
medium 1pt;padding:0cm 0cm 0cm
6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Doing
"optimistic caching" like you
suggest sounds very promising. A
way to regain more robustness
would be as follows.</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">If the
build fails while building the
libraries or the stage2 compiler,
this might be a false negative due
to the optimistic caching.
Therefore, evict the "optimistic
caches" and restart building the
libraries. That way we can
validate that the build failure
was a true build failure and not
just due to the aggressive caching
scheme.</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Just my 2p</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt">Josef</span></p>
</div>
<div>
<p class="MsoNormal"><span
style="font-size:12pt"> </span></p>
</div>
<div class="MsoNormal"
style="text-align:center"
align="center">
<hr width="98%" size="2"
align="center">
</div>
<div
id="gmail-m_6346696964856726737gmail-m_8288259843833037528divRplyFwdMsg">
<p class="MsoNormal"><b>From:</b>
ghc-devs <<a
href="mailto:ghc-devs-bounces@haskell.org"
target="_blank"
moz-do-not-send="true">ghc-devs-bounces@haskell.org</a>>
on behalf of Simon Peyton Jones via
ghc-devs <<a
href="mailto:ghc-devs@haskell.org"
target="_blank"
moz-do-not-send="true">ghc-devs@haskell.org</a>><br>
<b>Sent:</b> Friday, February 19,
2021 8:57 AM<br>
<b>To:</b> John Ericson <<a
href="mailto:john.ericson@obsidian.systems"
target="_blank"
moz-do-not-send="true">john.ericson@obsidian.systems</a>>;
ghc-devs <<a
href="mailto:ghc-devs@haskell.org"
target="_blank"
moz-do-not-send="true">ghc-devs@haskell.org</a>><br>
<b>Subject:</b> RE: On CI </p>
<div>
<p class="MsoNormal"> </p>
</div>
</div>
<div>
<div>
<ol style="margin-top:0cm" type="1"
start="1">
<li class="MsoNormal">Building and
testing happen together. When
tests failure spuriously, we
also have to rebuild GHC in
addition to re-running the
tests. That's pure waste.
<a
href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fissues%2F13897&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691140326%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Nm6vfgGLLlJpiGa8XKxI6kNkBetp8ZZLPZS8hF%2BydrM%3D&reserved=0"
target="_blank"
moz-do-not-send="true">
https://gitlab.haskell.org/ghc/ghc/-/issues/13897</a> tracks this more
or less.</li>
</ol>
<div>
<p class="MsoNormal">I don’t get
this. We have to build GHC
before we can test it, don’t we?</p>
</div>
<div style="margin-left:18pt">
<p class="MsoNormal">2 . We don't
cache between jobs. </p>
</div>
<div>
<p class="MsoNormal">This is, I
think, the big one. We
endlessly build the exact same
binaries.</p>
</div>
<div>
<p class="MsoNormal">There is a
problem, though. If we make *<b>any</b>*
change in GHC, even a trivial
refactoring, its binary will
change slightly. So now any
caching build system will assume
that anything built by that GHC
must be rebuilt – we can’t use
the cached version. That
includes all the libraries and
the stage2 compiler. So caching
can save all the preliminaries
(building the initial Cabal, and
large chunk of stage1, since
they are built with the same
bootstrap compiler) but after
that we are dead.</p>
</div>
<div>
<p class="MsoNormal">I don’t know
any robust way out of this.
That small change in the source
code of GHC might be trivial
refactoring, or it might
introduce a critical
mis-compilation which we really
want to see in its build
products.
</p>
</div>
<div>
<p class="MsoNormal">However, for
smoke-testing MRs, on every
architecture, we could perhaps
cut corners. (Leaving Marge to
do full diligence.) For
example, we could declare that
if we have the result of
compiling library module X.hs
with the stage1 GHC in the last
full commit in master, then we
can re-use that build product
rather than compiling X.hs with
the MR’s slightly modified
stage1 GHC. That *<b>might</b>*
be wrong; but it’s usually
right.</p>
</div>
<div>
<p class="MsoNormal">Anyway, there
are big wins to be had here.</p>
</div>
<div>
<p class="MsoNormal">Simon</p>
</div>
<p class="MsoNormal"
style="margin-left:18pt"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div style="border-style:none none
none solid;border-width:medium
medium medium 1.5pt;padding:0cm
0cm 0cm
4pt;border-color:currentcolor
currentcolor currentcolor blue">
<div>
<div style="border-style:solid
none none;border-width:1pt
medium medium;padding:3pt 0cm
0cm;border-color:currentcolor">
<div>
<p class="MsoNormal"><b><span
lang="EN-US">From:</span></b><span
lang="EN-US"> ghc-devs
<<a
href="mailto:ghc-devs-bounces@haskell.org"
target="_blank"
moz-do-not-send="true">ghc-devs-bounces@haskell.org</a>>
<b>On Behalf Of </b>John
Ericson<br>
<b>Sent:</b> 19 February
2021 03:19<br>
<b>To:</b> ghc-devs <<a
href="mailto:ghc-devs@haskell.org" target="_blank"
moz-do-not-send="true">ghc-devs@haskell.org</a>><br>
<b>Subject:</b> Re: On
CI</span></p>
</div>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal">I am also
wary of us to deferring checking
whole platforms and what not. I
think that's just kicking the
can down the road, and will
result in more variance and
uncertainty. It might be alright
for those authoring PRs, but it
will make Ben's job keeping the
system running even more
grueling.</p>
<p class="MsoNormal">Before
getting into these complex
trade-offs, I think we should
focus on the cornerstone issue
that CI isn't incremental.</p>
<ol style="margin-top:0cm"
type="1" start="1">
<li class="MsoNormal">Building
and testing happen together.
When tests failure spuriously,
we also have to rebuild GHC in
addition to re-running the
tests. That's pure waste.
<a
href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2F-%2Fissues%2F13897&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691150320%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=KlQGW1amK%2BtlRTGl4cDgMyl%2Bfz17fuUAHFNAaNXbzZI%3D&reserved=0"
target="_blank"
moz-do-not-send="true">
https://gitlab.haskell.org/ghc/ghc/-/issues/13897</a> tracks this more
or less.</li>
<li class="MsoNormal">We don't
cache between jobs. Shake and
Make do not enforce dependency
soundness, nor
cache-correctness when the
build plan itself changes, and
this had made this
hard/impossible to do safely.
Naively this only helps with
stage 1 and not stage 2, but
if we have separate stage 1
and --freeze1 stage 2 builds,
both can be incremental. Yes,
this is also lossy, but I only
see it leading to false
failures not false acceptances
(if we can also test the stage
1 one), so I consider it safe.
MRs that only work with a slow
full build because ABI can so
indicate.</li>
</ol>
<div>
<p class="MsoNormal">The second,
main part is quite hard to
tackle, but I strongly believe
incrementality is what we need
most, and what we should
remain focused on.</p>
</div>
<p class="MsoNormal">John</p>
</div>
</div>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
ghc-devs mailing list<br>
<a href="mailto:ghc-devs@haskell.org"
target="_blank" moz-do-not-send="true">ghc-devs@haskell.org</a><br>
<a
href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691160313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uE1IOblLTYJ2j3H2vkFKgQyVZs5sehXd1Tl70X0kUqE%3D&reserved=0"
target="_blank" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
ghc-devs mailing list<br>
<a href="mailto:ghc-devs@haskell.org"
target="_blank" moz-do-not-send="true">ghc-devs@haskell.org</a><br>
<a
href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691160313%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uE1IOblLTYJ2j3H2vkFKgQyVZs5sehXd1Tl70X0kUqE%3D&reserved=0"
target="_blank" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal"><br>
<br>
</p>
<pre>_______________________________________________</pre>
<pre>ghc-devs mailing list</pre>
<pre><a href="mailto:ghc-devs@haskell.org" target="_blank" moz-do-not-send="true">ghc-devs@haskell.org</a></pre>
<pre><a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fmail.haskell.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fghc-devs&data=04%7C01%7Csimonpj%40microsoft.com%7C9d7043627f5042598e5b08d8d6f648c4%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637495701691170308%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Yrob9grqAWOxZnFXcM%2BZ60VNsrhIejcmwkSIR3Wq0gA%3D&reserved=0" target="_blank" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a></pre>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
ghc-devs mailing list<br>
<a href="mailto:ghc-devs@haskell.org" target="_blank"
moz-do-not-send="true">ghc-devs@haskell.org</a><br>
<a
href="http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a><br>
</blockquote>
</div>
</blockquote>
</body>
</html>