Hadrian Transitive Dependencies

Wed Mar 27 16:00:34 UTC 2019

I also remember this discussion.. I think it was in the context of
optimising hadrian.

On Wed, 27 Mar 2019, 15:56 Alp Mestanogullari, <alp at well-typed.com> wrote:

>
> https://gitlab.haskell.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&search=%22ghc+-M%22
> seems to suggest this never made it into a ticket. I searched in the
> Hadrian github repo as well, no luck there either. It certainly came up in
> various discussions I've had though.
> On 27/03/2019 16:05, Andrey Mokhov wrote:
>
> Hi David,
>
>
>
> We had a discussion about this with Neil some time ago, and I think we had
> the following list of progressively more complex invariants for different
> types of build systems:
>
>
>
> ·        Non-cloud build systems: **all direct inputs must be declared**.
> If you miss a direct input dependency then a build may complete
> successfully but with an incorrect result.
>
>
>
> ·        Cloud build systems: **all direct inputs and direct outputs must
> be declared**. If you miss a direct output then a build may fail because
> the cloud will not be able to restore the corresponding output.
>
>
>
> ·        Cloud build systems with shallow (deferred) materialisation of
> build artefacts: **all transitive inputs and direct outputs must be
> declared**. Let’s say you’d like to download the resulting GHC binary
> directly, without materialising any intermediate artefacts. Then you’ll
> need to know GHC’s ultimate transitive inputs.
>
>
>
> I think for now we are really keen to make Hadrian a cloud build system,
> but whether shallow builds are valuable enough is not clear. Maybe not.
> Therefore, I’d say we don’t need to track transitive inputs right now.
> Furthermore, if we were to track all transitive inputs, we would lose the
> desirable early cutoff property, which prevents rebuilding after adding a
> comment in a file on which a lot of other files transitively depend on.
>
>
>
> Having said that, if we really access a file during compilation, then I
> think it is **not** a transitive dependency by definition! Any file which
> is accessed during a build rule is a direct dependency.
>
>
>
> > GHC is reading *.hi files that are not reported as dependencies by
>
> > `ghc -M  -include-pkg-deps`. This is because they are not direct, but
> transitive
>
> > dependencies!
>
>
>
> So, here I’m confused. If we read a file A when compiling a file B, then
> it’s by definition a direct dependency. Perhaps we just read too much?
> Maybe the solution is to switch to fine-grained `ghc -M` mode, to analyse
> import dependencies for a single module instead of doing it transitively,
> which I believe was discussed in a ticket some time ago? I can’t find this
> ticket, but I think Alp was looking into it at some point. Alp: do you
> remember it?
>
>
>
> Thank you for all your work on Hadrian!
>
>
>
> Cheers,
>
> Andrey
>
>
>
> *From:* David Eichmann [mailto:davide at well-typed.com
> <davide at well-typed.com>]
> *Sent:* 27 March 2019 12:54
> *To:* Neil Mitchell <ndmitchell at gmail.com> <ndmitchell at gmail.com>; Andrey
> Mokhov <andrey.mokhov at newcastle.ac.uk> <andrey.mokhov at newcastle.ac.uk>;
> GHC developers <ghc-devs at haskell.org> <ghc-devs at haskell.org>
> *Subject:* Hadrian Transitive Dependencies
>
>
>
> Hello Shake/Hadrian contributors and the like,
>
> Recently I've been putting Hadrian's fsatrace linting feature to good use,
> tracking down missing dependencies in Hadrian. Ultimately, we want to use
> shake's cloud build / shared cache feature and ensure it works across CI
> builds. Unfortunately the feature isn't working smoothly with Hadrian: see
> #16295 <https://gitlab.haskell.org/ghc/ghc/issues/16295>. This is very
> desirable to improve CI build times. It is my understanding that in order
> to get caching to work:
>
> 1. All accessed files must declared with `need` AND
> 2. All created files must be declared with `produces` (or be the target of
> the build rule)
>
> Is my understanding correct? Or is there a weaker condition (perhaps only
> 2 is necessary)?
>
> If I'm correct, this amounts to fixing all fsatrace lint errors. See here
> <https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901> for a
> breakdown of lint errors / missing dependencies. A large portion of these
> are Haskell interface files (i.e. *.hi files). Before building a Haskell
> object file, dependencies are discovered via `ghc` using the `-M
> -include-pkg-deps` options. Unfortunately, shake's fsatrace linting
> complains about other *.hi files being accessed! For example when building
> `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the following
> dependencies from ghc:
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi
>
> And shake complains of the following missing deps:
>
> _build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags
>
> Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon:
>
>   Used:  _build/HEAD_default/stage0/lib/settings
>
>   Used:  _build/HEAD_default/stage0/lib/platformConstants
>
>   Used:  _build/HEAD_default/stage0/lib/llvm-targets
>
>   Used:  _build/HEAD_default/stage0/lib/llvm-passes
>
>   Used:  _build/HEAD_default/stage0/lib/package.conf.d/package.cache
>
>   Used:  _build/HEAD_default/stage1/lib/package.conf.d/package.cache
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi
>
>   Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi
>
> GHC is reading *.hi files that are not reported as dependencies by `ghc
> -M  -include-pkg-deps`. This is because they are not direct, but
> *transitive* dependencies! How do we fix these lint errors (again with
> the goal of using shakes shared cache feature)? Some ideas:
>
> * Wildly over approximate dependencies. This may be easier to implement
> but cause unneeded recompilation (when a false dependency changes). Either:
>     * `need` all dependent packages' interface files recursively as well
> as transitive dependencies reported by `ghc -M  -include-pkg-deps` within
> the current package. OR
>     * OR `need` all transitive dependencies reported by `ghc -M
> -include-pkg-deps`. This will likely result in fewer dependencies but
> requires a bit more work in recovering dependent packages' dependency
> graphs.
> * Perhaps transitive dependencies are not important for shared caching to
> work. Change shakes linting feature to allow (untracked?) transitive
> dependencies to be accessed.
>
> Feed back would be greatly appreciated.
>
> David Eichmann
>
>
>
> --
>
> David Eichmann, Haskell Consultant
>
> Well-Typed LLP, http://www.well-typed.com
>
>
>
> Registered in England & Wales, OC335890
>
> 118 Wymering Mansions, Wymering Road, London W9 2NF, England
>
> --
> Alp Mestanogullari, Haskell Consultant
> Well-Typed LLP, https://www.well-typed.com/
>
> Registered in England and Wales, OC335890
> 118 Wymering Mansions, Wymering Road, London, W9 2NF, England
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20190327/d71bb62f/attachment.html>