<div dir="auto">I also remember this discussion.. I think it was in the context of optimising hadrian. </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 27 Mar 2019, 15:56 Alp Mestanogullari, <<a href="mailto:alp@well-typed.com">alp@well-typed.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><a class="m_-7768292169121230129moz-txt-link-freetext" href="https://gitlab.haskell.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&search=%22ghc+-M%22" target="_blank" rel="noreferrer">https://gitlab.haskell.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&search=%22ghc+-M%22</a>
seems to suggest this never made it into a ticket. I searched in
the Hadrian github repo as well, no luck there either. It
certainly came up in various discussions I've had though.<br>
</p>
<div class="m_-7768292169121230129moz-cite-prefix">On 27/03/2019 16:05, Andrey Mokhov
wrote:<br>
</div>
<blockquote type="cite">
<div class="m_-7768292169121230129WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Hi
David,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">We
had a discussion about this with Neil some time ago, and I
think we had the following list of progressively more
complex invariants for different types of build systems:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="m_-7768292169121230129MsoListParagraph"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Non-cloud
build systems: *<b>all direct inputs must be declared</b>*.
If you miss a direct input dependency then a build may
complete successfully but with an incorrect result.<u></u><u></u></span></p>
<p class="m_-7768292169121230129MsoListParagraph"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="m_-7768292169121230129MsoListParagraph"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cloud
build systems: *<b>all direct inputs and direct outputs must
be declared</b>*. If you miss a direct output then a build
may fail because the cloud will not be able to restore the
corresponding output.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="m_-7768292169121230129MsoListParagraph"><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cloud
build systems with shallow (deferred) materialisation of
build artefacts: *<b>all transitive inputs and direct
outputs must be declared</b>*. Let’s say you’d like to
download the resulting GHC binary directly, without
materialising any intermediate artefacts. Then you’ll need
to know GHC’s ultimate transitive inputs.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I
think for now we are really keen to make Hadrian a cloud
build system, but whether shallow builds are valuable enough
is not clear. Maybe not. Therefore, I’d say we don’t need to
track transitive inputs right now. Furthermore, if we were
to track all transitive inputs, we would lose the desirable
early cutoff property, which prevents rebuilding after
adding a comment in a file on which a lot of other files
transitively depend on.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Having
said that, if we really access a file during compilation,
then I think it is *<b>not</b>* a transitive dependency by
definition! Any file which is accessed during a build rule
is a direct dependency.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">>
GHC is reading *.hi files that are not reported as
dependencies by<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">>
`ghc -M -include-pkg-deps`. This is because they are not
direct, but transitive<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">>
dependencies!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">So,
here I’m confused. If we read a file A when compiling a file
B, then it’s by definition a direct dependency. Perhaps we
just read too much? Maybe the solution is to switch to
fine-grained `ghc -M` mode, to analyse import dependencies
for a single module instead of doing it transitively, which
I believe was discussed in a ticket some time ago? I can’t
find this ticket, but I think Alp was looking into it at
some point. Alp: do you remember it?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Thank
you for all your work on Hadrian!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Cheers,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Andrey<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_-7768292169121230129__MailEndCompose" rel="noreferrer"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></a></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext" lang="EN-US">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext" lang="EN-US"> David Eichmann
[<a class="m_-7768292169121230129moz-txt-link-freetext" href="mailto:davide@well-typed.com" target="_blank" rel="noreferrer">mailto:davide@well-typed.com</a>]
<br>
<b>Sent:</b> 27 March 2019 12:54<br>
<b>To:</b> Neil Mitchell <a class="m_-7768292169121230129moz-txt-link-rfc2396E" href="mailto:ndmitchell@gmail.com" target="_blank" rel="noreferrer"><ndmitchell@gmail.com></a>;
Andrey Mokhov <a class="m_-7768292169121230129moz-txt-link-rfc2396E" href="mailto:andrey.mokhov@newcastle.ac.uk" target="_blank" rel="noreferrer"><andrey.mokhov@newcastle.ac.uk></a>; GHC
developers <a class="m_-7768292169121230129moz-txt-link-rfc2396E" href="mailto:ghc-devs@haskell.org" target="_blank" rel="noreferrer"><ghc-devs@haskell.org></a><br>
<b>Subject:</b> Hadrian Transitive Dependencies<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p>Hello Shake/Hadrian contributors and the like,<u></u><u></u></p>
<p>Recently I've been putting Hadrian's fsatrace linting feature
to good use, tracking down missing dependencies in Hadrian.
Ultimately, we want to use shake's cloud build / shared cache
feature and ensure it works across CI builds. Unfortunately
the feature isn't working smoothly with Hadrian: see <a href="https://gitlab.haskell.org/ghc/ghc/issues/16295" target="_blank" rel="noreferrer">
#16295</a>. This is very desirable to improve CI build
times. It is my understanding that in order to get caching to
work:<u></u><u></u></p>
<p class="MsoNormal">1. All accessed files must declared with
`need` AND<br>
2. All created files must be declared with `produces` (or be
the target of the build rule)
<u></u><u></u></p>
<p>Is my understanding correct? Or is there a weaker condition
(perhaps only 2 is necessary)?<u></u><u></u></p>
<p>If I'm correct, this amounts to fixing all fsatrace lint
errors. See <a href="https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901" target="_blank" rel="noreferrer">
here</a> for a breakdown of lint errors / missing
dependencies. A large portion of these are Haskell interface
files (i.e. *.hi files). Before building a Haskell object
file, dependencies are discovered via `ghc` using the `-M
-include-pkg-deps` options. Unfortunately, shake's fsatrace
linting complains about other *.hi files being accessed! For
example when building
`stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get
the following dependencies from ghc:<u></u><u></u></p>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi<u></u><u></u></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi<u></u><u></u></pre>
<p>And shake complains of the following missing deps:<u></u><u></u></p>
<pre>_build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags<u></u><u></u></pre>
<pre>Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon:<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage0/lib/settings<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage0/lib/platformConstants<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage0/lib/llvm-targets<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage0/lib/llvm-passes<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi<u></u><u></u></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi<u></u><u></u></pre>
<p>GHC is reading *.hi files that are not reported as
dependencies by `ghc -M -include-pkg-deps`. This is because
they are not direct, but
<i>transitive</i> dependencies! How do we fix these lint
errors (again with the goal of using shakes shared cache
feature)? Some ideas:<u></u><u></u></p>
<p class="MsoNormal">* Wildly over approximate dependencies.
This may be easier to implement but cause unneeded
recompilation (when a false dependency changes). Either:<br>
* `need` all dependent packages' interface files
recursively as well as transitive dependencies reported by
`ghc -M -include-pkg-deps` within the current package. OR<br>
* OR `need` all transitive dependencies reported by `ghc
-M -include-pkg-deps`. This will likely result in fewer
dependencies but requires a bit more work in recovering
dependent packages' dependency graphs.<br>
* Perhaps transitive dependencies are not important for shared
caching to work. Change shakes linting feature to allow
(untracked?) transitive dependencies to be accessed.
<u></u><u></u></p>
<p>Feed back would be greatly appreciated.<u></u><u></u></p>
<p>David Eichmann<u></u><u></u></p>
<p><u></u> <u></u></p>
<pre>-- <u></u><u></u></pre>
<pre>David Eichmann, Haskell Consultant<u></u><u></u></pre>
<pre>Well-Typed LLP, <a href="http://www.well-typed.com" target="_blank" rel="noreferrer">http://www.well-typed.com</a><u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Registered in England & Wales, OC335890<u></u><u></u></pre>
<pre>118 Wymering Mansions, Wymering Road, London W9 2NF, England <u></u><u></u></pre>
</div>
</blockquote>
<pre class="m_-7768292169121230129moz-signature" cols="72">--
Alp Mestanogullari, Haskell Consultant
Well-Typed LLP, <a class="m_-7768292169121230129moz-txt-link-freetext" href="https://www.well-typed.com/" target="_blank" rel="noreferrer">https://www.well-typed.com/</a>
Registered in England and Wales, OC335890
118 Wymering Mansions, Wymering Road, London, W9 2NF, England</pre>
</div>
_______________________________________________<br>
ghc-devs mailing list<br>
<a href="mailto:ghc-devs@haskell.org" target="_blank" rel="noreferrer">ghc-devs@haskell.org</a><br>
<a href="http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs" rel="noreferrer noreferrer" target="_blank">http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</a><br>
</blockquote></div>