<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hello Shake/Hadrian contributors and the like,</p>
<p>Recently I've been putting Hadrian's fsatrace linting feature to
good use, tracking down missing dependencies in Hadrian.
Ultimately, we want to use shake's cloud build / shared cache
feature and ensure it works across CI builds. Unfortunately the
feature isn't working smoothly with Hadrian: see <a
moz-do-not-send="true"
href="https://gitlab.haskell.org/ghc/ghc/issues/16295">#16295</a>.
This is very desirable to improve CI build times. It is my
understanding that in order to get caching to work:</p>
1. All accessed files must declared with `need` AND<br>
2. All created files must be declared with `produces` (or be the
target of the build rule)
<p>Is my understanding correct? Or is there a weaker condition
(perhaps only 2 is necessary)?<br>
</p>
<p>If I'm correct, this amounts to fixing all fsatrace lint errors.
See <a moz-do-not-send="true"
href="https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901">here</a>
for a breakdown of lint errors / missing dependencies. A large
portion of these are Haskell interface files (i.e. *.hi files).
Before building a Haskell object file, dependencies are discovered
via `ghc` using the `-M -include-pkg-deps` options.
Unfortunately, shake's fsatrace linting complains about other *.hi
files being accessed! For example when building
`stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the
following dependencies from ghc:</p>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi
_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi</pre>
<p> And shake complains of the following missing deps:</p>
<pre>_build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags
Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon:
Used: _build/HEAD_default/stage0/lib/settings
Used: _build/HEAD_default/stage0/lib/platformConstants
Used: _build/HEAD_default/stage0/lib/llvm-targets
Used: _build/HEAD_default/stage0/lib/llvm-passes
Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache
Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi
Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi
</pre>
<p>GHC is reading *.hi files that are not reported as dependencies
by `ghc -M -include-pkg-deps`. This is because they are not
direct, but <i>transitive</i> dependencies! How do we fix these
lint errors (again with the goal of using shakes shared cache
feature)? Some ideas:</p>
* Wildly over approximate dependencies. This may be easier to
implement but cause unneeded recompilation (when a false dependency
changes). Either:<br>
* `need` all dependent packages' interface files recursively as
well as transitive dependencies reported by `ghc -M
-include-pkg-deps` within the current package. OR<br>
* OR `need` all transitive dependencies reported by `ghc -M
-include-pkg-deps`. This will likely result in fewer dependencies
but requires a bit more work in recovering dependent packages'
dependency graphs.<br>
* Perhaps transitive dependencies are not important for shared
caching to work. Change shakes linting feature to allow (untracked?)
transitive dependencies to be accessed.
<p>Feed back would be greatly appreciated.</p>
<p>David Eichmann<br>
</p>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
David Eichmann, Haskell Consultant
Well-Typed LLP, <a class="moz-txt-link-freetext" href="http://www.well-typed.com">http://www.well-typed.com</a>
Registered in England & Wales, OC335890
118 Wymering Mansions, Wymering Road, London W9 2NF, England </pre>
</body>
</html>