<div dir="ltr"><div><div><div>How's this? <a href="https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter">https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter</a><br><br></div>I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code. There are already a few Notes (e.g. <a href="https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86">https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86</a>) but if anything is confusing I'll happily add more Notes.<br><br></div>Cheers<br></div>Simon<br><div><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On 22 June 2016 at 11:50, Simon Marlow <span dir="ltr"><<a href="mailto:marlowsd@gmail.com" target="_blank">marlowsd@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="">On 22 June 2016 at 11:37, Simon Peyton Jones <span dir="ltr"><<a href="mailto:simonpj@microsoft.com" target="_blank">simonpj@microsoft.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-GB">
<div>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">I’m ok with this.
<i>It would certainly be great not to support TWO mechanisms indefinitely</i>.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">What are the disadvantages to committing to this path? Would anyone even notice?<br></span></p></div></div></blockquote><div><br></div></span><div>Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter. In some cases that might mean extra work, e.g. if we do <a href="https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective" target="_blank">TemplateHaskell/Introspective</a> then potentially the whole of HsSyn needs to be binary serializable. That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.<br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div link="blue" vlink="purple" lang="EN-GB"><div><p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works. I would love to see an implementation overview, either
somewhere in the code or on a wiki page. Things like:<u></u><u></u></span></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-family:"Calibri",sans-serif">How, when, and where in the compiler is the separate process started?<u></u><u></u></span></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-family:"Calibri",sans-serif">How do the compiler and server communicate? Unix pipes? Is it the same on Windows and Unix?<u></u><u></u></span></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-family:"Calibri",sans-serif">What is serialised, when, and how? For example, GHC has some TH code to run. Do we send a syntax tree? Or compile to bytecode and send that? Or what?<u></u><u></u></span></p>
<p><u></u><span style="font-family:Symbol"><span>·<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-family:"Calibri",sans-serif">How are external references managed. E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called
or where it lives.</span></p></div></div></blockquote><div><br></div></span><div>Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.<br><br></div><div>Cheers<span class=""><font color="#888888"><br></font></span></div><span class=""><font color="#888888"><div>Simon<br></div></font></span><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div link="blue" vlink="purple" lang="EN-GB"><div><p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">Thanks<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif">Simon<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div style="border-width:medium medium medium 1.5pt;border-style:none none none solid;border-color:-moz-use-text-color -moz-use-text-color -moz-use-text-color blue;padding:0cm 0cm 0cm 4pt">
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(225,225,225) -moz-use-text-color -moz-use-text-color;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> ghc-devs [mailto:<a href="mailto:ghc-devs-bounces@haskell.org" target="_blank">ghc-devs-bounces@haskell.org</a>]
<b>On Behalf Of </b>Simon Marlow<br>
<b>Sent:</b> 22 June 2016 09:51<br>
<b>To:</b> <a href="mailto:ghc-devs@haskell.org" target="_blank">ghc-devs@haskell.org</a><br>
<b>Subject:</b> Require -fexternal-interpreter support for future TH changes?<u></u><u></u></span></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:12pt;margin-left:0cm">
<b>Background</b><u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-right:0cm;margin-bottom:6pt;margin-left:0cm">
A few months ago I added -fexternal-interpreter to GHC: <u></u><u></u></p>
<ul type="disc">
<li class="MsoNormal">
docs: <a href="https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d" target="_blank">
http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter</a><u></u><u></u></li><li class="MsoNormal">
wiki, rationale: <a href="https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi" target="_blank">https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi</a><u></u><u></u></li></ul>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar
Christina.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><b>Proposal</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Why?<u></u><u></u></p>
</div>
<ul type="disc">
<li class="MsoNormal">
-fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS.<u></u><u></u></li><li class="MsoNormal">
We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more.<u></u><u></u></li><li class="MsoNormal">
Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.<u></u><u></u></li></ul>
<div>
<p class="MsoNormal">How?<u></u><u></u></p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal">
I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.<u></u><u></u></li></ul>
<p><b>Why now?</b><u></u><u></u></p>
<p>There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g.
<a href="https://phabricator.haskell.org/D2286" target="_blank">https://phabricator.haskell.org/D2286</a> and
<a href="https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective" target="_blank">https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective</a>, so I'd like to raise it now so we can keep the issue in mind.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Cheers<u></u><u></u></p>
<p>Simon<u></u><u></u></p>
</div>
</div>
</div></div></div>
</div>
</div>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div></div></div></div></div>