Hadrian Transitive Dependencies

Wed Mar 27 15:56:34 UTC 2019

https://gitlab.haskell.org/dashboard/issues?scope=all&utf8=%E2%9C%93&state=opened&search=%22ghc+-M%22 
seems to suggest this never made it into a ticket. I searched in the 
Hadrian github repo as well, no luck there either. It certainly came up 
in various discussions I've had though.

On 27/03/2019 16:05, Andrey Mokhov wrote:
>
> Hi David,
>
> We had a discussion about this with Neil some time ago, and I think we 
> had the following list of progressively more complex invariants for 
> different types of build systems:
>
> ·Non-cloud build systems: **all direct inputs must be declared**. If 
> you miss a direct input dependency then a build may complete 
> successfully but with an incorrect result.
>
> ·Cloud build systems: **all direct inputs and direct outputs must be 
> declared**. If you miss a direct output then a build may fail because 
> the cloud will not be able to restore the corresponding output.
>
> ·Cloud build systems with shallow (deferred) materialisation of build 
> artefacts: **all transitive inputs and direct outputs must be 
> declared**. Let’s say you’d like to download the resulting GHC binary 
> directly, without materialising any intermediate artefacts. Then 
> you’ll need to know GHC’s ultimate transitive inputs.
>
> I think for now we are really keen to make Hadrian a cloud build 
> system, but whether shallow builds are valuable enough is not clear. 
> Maybe not. Therefore, I’d say we don’t need to track transitive inputs 
> right now. Furthermore, if we were to track all transitive inputs, we 
> would lose the desirable early cutoff property, which prevents 
> rebuilding after adding a comment in a file on which a lot of other 
> files transitively depend on.
>
> Having said that, if we really access a file during compilation, then 
> I think it is **not** a transitive dependency by definition! Any file 
> which is accessed during a build rule is a direct dependency.
>
> > GHC is reading *.hi files that are not reported as dependencies by
>
> > `ghc -M  -include-pkg-deps`. This is because they are not direct, but 
> transitive
>
> > dependencies!
>
> So, here I’m confused. If we read a file A when compiling a file B, 
> then it’s by definition a direct dependency. Perhaps we just read too 
> much? Maybe the solution is to switch to fine-grained `ghc -M` mode, 
> to analyse import dependencies for a single module instead of doing it 
> transitively, which I believe was discussed in a ticket some time ago? 
> I can’t find this ticket, but I think Alp was looking into it at some 
> point. Alp: do you remember it?
>
> Thank you for all your work on Hadrian!
>
> Cheers,
>
> Andrey
>
> *From:*David Eichmann [mailto:davide at well-typed.com]
> *Sent:* 27 March 2019 12:54
> *To:* Neil Mitchell <ndmitchell at gmail.com>; Andrey Mokhov 
> <andrey.mokhov at newcastle.ac.uk>; GHC developers <ghc-devs at haskell.org>
> *Subject:* Hadrian Transitive Dependencies
>
> Hello Shake/Hadrian contributors and the like,
>
> Recently I've been putting Hadrian's fsatrace linting feature to good 
> use, tracking down missing dependencies in Hadrian. Ultimately, we 
> want to use shake's cloud build / shared cache feature and ensure it 
> works across CI builds. Unfortunately the feature isn't working 
> smoothly with Hadrian: see #16295 
> <https://gitlab.haskell.org/ghc/ghc/issues/16295>. This is very 
> desirable to improve CI build times. It is my understanding that in 
> order to get caching to work:
>
> 1. All accessed files must declared with `need` AND
> 2. All created files must be declared with `produces` (or be the 
> target of the build rule)
>
> Is my understanding correct? Or is there a weaker condition (perhaps 
> only 2 is necessary)?
>
> If I'm correct, this amounts to fixing all fsatrace lint errors. See 
> here <https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901> for 
> a breakdown of lint errors / missing dependencies. A large portion of 
> these are Haskell interface files (i.e. *.hi files). Before building a 
> Haskell object file, dependencies are discovered via `ghc` using the 
> `-M -include-pkg-deps` options. Unfortunately, shake's fsatrace 
> linting complains about other *.hi files being accessed! For example 
> when building `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` 
> we get the following dependencies from ghc:
>
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi
> _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi
>
> And shake complains of the following missing deps:
>
> _build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags
> Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon:
>    Used:  _build/HEAD_default/stage0/lib/settings
>    Used:  _build/HEAD_default/stage0/lib/platformConstants
>    Used:  _build/HEAD_default/stage0/lib/llvm-targets
>    Used:  _build/HEAD_default/stage0/lib/llvm-passes
>    Used:  _build/HEAD_default/stage0/lib/package.conf.d/package.cache
>    Used:  _build/HEAD_default/stage1/lib/package.conf.d/package.cache
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi
>    Used:  _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi
>
> GHC is reading *.hi files that are not reported as dependencies by 
> `ghc -M  -include-pkg-deps`. This is because they are not direct, but 
> /transitive/ dependencies! How do we fix these lint errors (again with 
> the goal of using shakes shared cache feature)? Some ideas:
>
> * Wildly over approximate dependencies. This may be easier to 
> implement but cause unneeded recompilation (when a false dependency 
> changes). Either:
>     * `need` all dependent packages' interface files recursively as 
> well as transitive dependencies reported by `ghc -M  
> -include-pkg-deps` within the current package. OR
>     * OR `need` all transitive dependencies reported by `ghc -M  
> -include-pkg-deps`. This will likely result in fewer dependencies but 
> requires a bit more work in recovering dependent packages' dependency 
> graphs.
> * Perhaps transitive dependencies are not important for shared caching 
> to work. Change shakes linting feature to allow (untracked?) 
> transitive dependencies to be accessed.
>
> Feed back would be greatly appreciated.
>
> David Eichmann
>
> -- 
> David Eichmann, Haskell Consultant
> Well-Typed LLP,http://www.well-typed.com
> Registered in England & Wales, OC335890
> 118 Wymering Mansions, Wymering Road, London W9 2NF, England

-- 
Alp Mestanogullari, Haskell Consultant
Well-Typed LLP, https://www.well-typed.com/

Registered in England and Wales, OC335890
118 Wymering Mansions, Wymering Road, London, W9 2NF, England

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20190327/707c8ebb/attachment.html>