<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
p.Code, li.Code, div.Code
{mso-style-name:Code;
mso-style-link:"Code Char";
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Courier New";
color:black;}
span.CodeChar
{mso-style-name:"Code Char";
mso-style-link:Code;
font-family:"Courier New";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle29
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:21907606;
mso-list-type:hybrid;
mso-list-template-ids:-1565080408 1348526704 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:16;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:Calibri;
mso-bidi-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:111097865;
mso-list-type:hybrid;
mso-list-template-ids:-914838060 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2
{mso-list-id:345521030;
mso-list-type:hybrid;
mso-list-template-ids:-1853178420 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3
{mso-list-id:1963002119;
mso-list-type:hybrid;
mso-list-template-ids:493916712 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:38.5pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:74.5pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:110.5pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:146.5pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:182.5pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:218.5pt;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:254.5pt;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:290.5pt;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:326.5pt;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Having said that, if we really access a file during compilation, then I think it is *<b>not</b>* a transitive
dependency by definition! Any file which is accessed during a build rule is a direct dependency.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">Well, ok, then we need a new definition! Suppose<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:windowtext;margin-left:0cm;mso-list:l1 level1 lfo3">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Module M imports A<o:p></o:p></span></li><li class="MsoListParagraph" style="color:windowtext;margin-left:0cm;mso-list:l1 level1 lfo3">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">A.hi mentions (in a type or unfolding) B.T<o:p></o:p></span></li><li class="MsoListParagraph" style="color:windowtext;margin-left:0cm;mso-list:l1 level1 lfo3">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Hence, to find out about B.T we need to read B.hi<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">So you would say that B is a direct dependency of M. Fine. I’ll call B.hi a “<b>deep dependency</b>” of M, but A.hi is a “<b>shallow
dependency</b>” of M.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">But it’s very hard to predict deep dependencies. Whether M deep-depends on B.hi depends on<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:windowtext;margin-left:2.5pt;mso-list:l3 level1 lfo4">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">the
<b>contents</b> of A.hi<o:p></o:p></span></li><li class="MsoListParagraph" style="color:windowtext;margin-left:2.5pt;mso-list:l3 level1 lfo4">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">the
<b>optimisation level</b> at which M is compiled<o:p></o:p></span></li><li class="MsoListParagraph" style="color:windowtext;margin-left:2.5pt;mso-list:l3 level1 lfo4">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">perhaps even the contents of some apparently
<b>unrelated</b> X.hi, directly imported by M, which influences the extent to which GHC’s optimiser dives into A’s exprots<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">So no ghc -M is going to do the job. To get the real answer you must compile M!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">Happily,
<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:windowtext;margin-left:0cm;mso-list:l2 level1 lfo5">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">we already list the deep dependencies in M.hi<o:p></o:p></span></li><li class="MsoListParagraph" style="color:windowtext;margin-left:0cm;mso-list:l2 level1 lfo5">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">and, as I mentioned earlier, if a deep dependency changes (such as B.hi) then GHC guarantees that a shallow dependency (such as A.hi) will change.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">Does that help?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US">Simon<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"> ghc-devs <ghc-devs-bounces@haskell.org>
<b>On Behalf Of </b>Andrey Mokhov<br>
<b>Sent:</b> 27 March 2019 15:06<br>
<b>To:</b> David Eichmann <davide@well-typed.com>; Neil Mitchell <ndmitchell@gmail.com><br>
<b>Cc:</b> GHC developers <ghc-devs@haskell.org><br>
<b>Subject:</b> RE: Hadrian Transitive Dependencies<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi David,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We had a discussion about this with Neil some time ago, and I think we had the following list of progressively more complex invariants
for different types of build systems:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:#1F497D;margin-left:0cm;mso-list:l0 level1 lfo2">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Non-cloud build systems: *<b>all direct inputs must be declared</b>*. If you miss a direct input dependency then a build may complete successfully but with an incorrect
result.<o:p></o:p></span></li></ul>
<p class="MsoListParagraph"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:#1F497D;margin-left:0cm;mso-list:l0 level1 lfo2">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Cloud build systems: *<b>all direct inputs and direct outputs must be declared</b>*. If you miss a direct output then a build may fail because the cloud will not be able
to restore the corresponding output.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="color:#1F497D;margin-left:0cm;mso-list:l0 level1 lfo2">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Cloud build systems with shallow (deferred) materialisation of build artefacts: *<b>all transitive inputs and direct outputs must be declared</b>*. Let’s say you’d like
to download the resulting GHC binary directly, without materialising any intermediate artefacts. Then you’ll need to know GHC’s ultimate transitive inputs.<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I think for now we are really keen to make Hadrian a cloud build system, but whether shallow builds are valuable enough is not clear.
Maybe not. Therefore, I’d say we don’t need to track transitive inputs right now. Furthermore, if we were to track all transitive inputs, we would lose the desirable early cutoff property, which prevents rebuilding after adding a comment in a file on which
a lot of other files transitively depend on.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Having said that, if we really access a file during compilation, then I think it is *<b>not</b>* a transitive dependency by definition!
Any file which is accessed during a build rule is a direct dependency.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">> GHC is reading *.hi files that are not reported as dependencies by<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">> `ghc -M -include-pkg-deps`. This is because they are not direct, but transitive<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">> dependencies!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">So, here I’m confused. If we read a file A when compiling a file B, then it’s by definition a direct dependency. Perhaps we just read
too much? Maybe the solution is to switch to fine-grained `ghc -M` mode, to analyse import dependencies for a single module instead of doing it transitively, which I believe was discussed in a ticket some time ago? I can’t find this ticket, but I think Alp
was looking into it at some point. Alp: do you remember it?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Thank you for all your work on Hadrian!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Andrey<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:windowtext"> David Eichmann [<a href="mailto:davide@well-typed.com">mailto:davide@well-typed.com</a>]
<br>
<b>Sent:</b> 27 March 2019 12:54<br>
<b>To:</b> Neil Mitchell <<a href="mailto:ndmitchell@gmail.com">ndmitchell@gmail.com</a>>; Andrey Mokhov <<a href="mailto:andrey.mokhov@newcastle.ac.uk">andrey.mokhov@newcastle.ac.uk</a>>; GHC developers <<a href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a>><br>
<b>Subject:</b> Hadrian Transitive Dependencies<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Hello Shake/Hadrian contributors and the like,<o:p></o:p></p>
<p>Recently I've been putting Hadrian's fsatrace linting feature to good use, tracking down missing dependencies in Hadrian. Ultimately, we want to use shake's cloud build / shared cache feature and ensure it works across CI builds. Unfortunately the feature
isn't working smoothly with Hadrian: see <a href="https://gitlab.haskell.org/ghc/ghc/issues/16295">
#16295</a>. This is very desirable to improve CI build times. It is my understanding that in order to get caching to work:<o:p></o:p></p>
<p class="MsoNormal">1. All accessed files must declared with `need` AND<br>
2. All created files must be declared with `produces` (or be the target of the build rule)
<o:p></o:p></p>
<p>Is my understanding correct? Or is there a weaker condition (perhaps only 2 is necessary)?<o:p></o:p></p>
<p>If I'm correct, this amounts to fixing all fsatrace lint errors. See <a href="https://gitlab.haskell.org/ghc/ghc/issues/16400#note_188901">
here</a> for a breakdown of lint errors / missing dependencies. A large portion of these are Haskell interface files (i.e. *.hi files). Before building a Haskell object file, dependencies are discovered via `ghc` using the `-M -include-pkg-deps` options. Unfortunately,
shake's fsatrace linting complains about other *.hi files being accessed! For example when building `stage1/libraries/mtl/build/Control/Monad/RWS/Class.o` we get the following dependencies from ghc:<o:p></o:p></p>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : libraries/mtl/Control/Monad/RWS/Class.hs<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Prelude.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Monoid.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Strict.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/RWS/Lazy.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Identity.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Maybe.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Except.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Error.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/lib/../lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Class.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Writer/Class.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/State/Class.hi<o:p></o:p></pre>
<pre>_build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o : _build/stage1/libraries/mtl/build/Control/Monad/Reader/Class.hi<o:p></o:p></pre>
<p>And shake complains of the following missing deps:<o:p></o:p></p>
<pre>_build/stage0/bin/ghc -Wall -hisuf hi -osuf o -hcsuf hc -static -hide-all-packages -no-user-package-db '-package-db _build/stage1/lib/package.conf.d' '-this-unit-id mtl-2.2.2' '-package-id base-4.13.0.0' '-package-id transformers-0.5.5.0' -i -i_build/stage1/libraries/mtl/build -i_build/stage1/libraries/mtl/build/autogen -ilibraries/mtl/. -Iincludes -I_build/generated -I_build/stage1/libraries/mtl/build -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/include -I/home/david/ghc/_build/stage1/lib/x86_64-linux-ghc-8.9.20190325/rts-1.0/include -I_build/generated -optc-I_build/generated -optP-include -optP_build/stage1/libraries/mtl/build/autogen/cabal_macros.h -outputdir _build/stage1/libraries/mtl/build -Wnoncanonical-monad-instances -optc-Werror=unused-but-set-variable -optc-Wno-error=inline -c libraries/mtl/Control/Monad/RWS/Class.hs -o _build/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o -O2 -H32m -Wall -fno-warn-unused-imports -fno-warn-warnings-deprecations -Wcompat -Wnoncanonical-monad-instances -Wnoncanonical-monadfail-instances -XHaskell2010 -XSafe -ghcversion-file=/home/david/MEGA/File_Dump/Well-Typed/GHC/_nosync_git/ghc/_build/generated/ghcversion.h -Wno-deprecated-flags<o:p></o:p></pre>
<pre>Lint checking error - _build/HEAD_default/stage1/libraries/mtl/build/Control/Monad/RWS/Class.o - 22 values were used but not depended upon:<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage0/lib/settings<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage0/lib/platformConstants<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage0/lib/llvm-targets<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage0/lib/llvm-passes<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage0/lib/package.conf.d/package.cache<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/package.conf.d/package.cache<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Float.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Base.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Types.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Maybe.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Lazy.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Writer/Strict.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Lazy.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/State/Strict.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Reader.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/List.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/transformers-0.5.5.0/Control/Monad/Trans/Cont.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/ghc-prim-0.5.3/GHC/Tuple.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/IO/Exception.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/integer-gmp-1.0.2.0/GHC/Integer/Type.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/Data/Either.hi<o:p></o:p></pre>
<pre> Used: _build/HEAD_default/stage1/lib/x86_64-linux-ghc-8.9.20190325/base-4.13.0.0/GHC/Natural.hi<o:p></o:p></pre>
<p>GHC is reading *.hi files that are not reported as dependencies by `ghc -M -include-pkg-deps`. This is because they are not direct, but
<i>transitive</i> dependencies! How do we fix these lint errors (again with the goal of using shakes shared cache feature)? Some ideas:<o:p></o:p></p>
<p class="MsoNormal">* Wildly over approximate dependencies. This may be easier to implement but cause unneeded recompilation (when a false dependency changes). Either:<br>
* `need` all dependent packages' interface files recursively as well as transitive dependencies reported by `ghc -M -include-pkg-deps` within the current package. OR<br>
* OR `need` all transitive dependencies reported by `ghc -M -include-pkg-deps`. This will likely result in fewer dependencies but requires a bit more work in recovering dependent packages' dependency graphs.<br>
* Perhaps transitive dependencies are not important for shared caching to work. Change shakes linting feature to allow (untracked?) transitive dependencies to be accessed.
<o:p></o:p></p>
<p>Feed back would be greatly appreciated.<o:p></o:p></p>
<p>David Eichmann<o:p></o:p></p>
<p><o:p> </o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>David Eichmann, Haskell Consultant<o:p></o:p></pre>
<pre>Well-Typed LLP, <a href="http://www.well-typed.com">http://www.well-typed.com</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Registered in England & Wales, OC335890<o:p></o:p></pre>
<pre>118 Wymering Mansions, Wymering Road, London W9 2NF, England <o:p></o:p></pre>
</div>
</div>
</body>
</html>