<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div>It seems there are nice advantages to this approach, and so I'm fine requiring support from TH. But here is a comment I just posted on D2286 requesting more comments in the code:</div><div><br></div><div>---</div><div><span class="transaction-comment" data-sigil="transaction-comment" data-meta="0_100">I've
just spent some time staring at GHCi/TH.hs and friends, and it's
starting to make sense. Would it be possible to add some comments/Notes
there? It would be great to know what is run on the client (GHC, right?)
and what on the server (TH, right?). Also, why do we need a <tt class="remarkup-monospaced">RemoteRef</tt> of an <tt class="remarkup-monospaced">IORef</tt>?
This, at first, strikes me as odd. I'm sure it makes great sense once I
understand all the other moving parts, but it would be helpful to have a
primer. I also find it a bit confusing that the <tt class="remarkup-monospaced">Message</tt> type contains both messages to the server and messages from the server. For example, I would expect <tt class="remarkup-monospaced">StartTH</tt> and <tt class="remarkup-monospaced">Reify</tt> to live in different datatypes, because the two should never mix. Or am I horribly mistaken?</span></div>---<div><br></div><div>Before we require -fexternal-interpreter (which seems to be the endgoal here), has anyone looked at performance implications? All the marshaling and unmarshaling has to take a toll.</div><div><br></div><div>Richard</div><div><br><div><div>On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs <<a href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-GB" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">It’s a great start, thanks<o:p></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; "> </span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">Simon<o:p></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; "> </span></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0cm 0cm 0cm 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0cm 0cm; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; ">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; "><span class="Apple-converted-space"> </span>Simon Marlow [mailto:marlowsd@<a href="http://gmail.com">gmail.com</a>]<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>22 June 2016 12:32<br><b>To:</b><span class="Apple-converted-space"> </span>Simon Peyton Jones <<a href="mailto:simonpj@microsoft.com">simonpj@microsoft.com</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: Require -fexternal-interpreter support for future TH changes?<o:p></o:p></span></div></div></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div><div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">How's this?<span class="Apple-converted-space"> </span><a href="https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter" style="color: purple; text-decoration: underline; ">https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter</a><o:p></o:p></p></div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code. There are already a few Notes (e.g.<a href="https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86" style="color: purple; text-decoration: underline; ">https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86</a>) but if anything is confusing I'll happily add more Notes.<o:p></o:p></p></div><p class="MsoNormal" style="margin: 0cm 0cm 6pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Cheers<o:p></o:p></p></div><p class="MsoNormal" style="margin: 0cm 0cm 6pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Simon<o:p></o:p></p><div><p class="MsoNormal" style="margin: 0cm 0cm 6pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></p><div><p class="MsoNormal" style="margin: 0cm 0cm 6pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">On 22 June 2016 at 11:50, Simon Marlow <<a href="mailto:marlowsd@gmail.com" target="_blank" style="color: purple; text-decoration: underline; ">marlowsd@gmail.com</a>> wrote:<o:p></o:p></p><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm; "><div><p class="MsoNormal" style="margin: 0cm 0cm 6pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">On 22 June 2016 at 11:37, Simon Peyton Jones <<a href="mailto:simonpj@microsoft.com" target="_blank" style="color: purple; text-decoration: underline; ">simonpj@microsoft.com</a>> wrote:<o:p></o:p></p><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">I’m ok with this. <span class="Apple-converted-space"> </span><i>It would certainly be great not to support TWO mechanisms indefinitely</i>.</span><o:p></o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; "> </span><o:p></o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">What are the disadvantages to committing to this path? Would anyone even notice?</span><o:p></o:p></div></blockquote><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter. In some cases that might mean extra work, e.g. if we do<span class="Apple-converted-space"> </span><a href="https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective" target="_blank" style="color: purple; text-decoration: underline; ">TemplateHaskell/Introspective</a><span class="Apple-converted-space"> </span>then potentially the whole of HsSyn needs to be binary serializable. That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works. I would love to see an implementation overview, either somewhere in the code or on a wiki page. Things like:</span><o:p></o:p></div><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Symbol; ">·</span><span style="font-size: 7pt; "> <span class="Apple-converted-space"> </span></span><span style="font-family: Calibri, sans-serif; ">How, when, and where in the compiler is the separate process started?</span><o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Symbol; ">·</span><span style="font-size: 7pt; "> <span class="Apple-converted-space"> </span></span><span style="font-family: Calibri, sans-serif; ">How do the compiler and server communicate? Unix pipes? Is it the same on Windows and Unix?</span><o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Symbol; ">·</span><span style="font-size: 7pt; "> <span class="Apple-converted-space"> </span></span><span style="font-family: Calibri, sans-serif; ">What is serialised, when, and how? For example, GHC has some TH code to run. Do we send a syntax tree? Or compile to bytecode and send that? Or what?</span><o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Symbol; ">·</span><span style="font-size: 7pt; "> <span class="Apple-converted-space"> </span></span><span style="font-family: Calibri, sans-serif; ">How are external references managed. E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.</span><o:p></o:p></p><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.</span><o:p></o:p></div></blockquote><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.<o:p></o:p></p></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Cheers<o:p></o:p></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="color: rgb(136, 136, 136); ">Simon<o:p></o:p></span></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0cm 0cm 0cm 6pt; margin-left: 4.8pt; margin-right: 0cm; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; "> </span><o:p></o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">Thanks</span><o:p></o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; "> </span><o:p></o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; ">Simon</span><o:p></o:p></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-family: Calibri, sans-serif; "> </span><o:p></o:p></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: windowtext; padding: 0cm 0cm 0cm 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: windowtext; padding: 3pt 0cm 0cm; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; ">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif; "><span class="Apple-converted-space"> </span>ghc-devs [mailto:<a href="mailto:ghc-devs-bounces@haskell.org" target="_blank" style="color: purple; text-decoration: underline; ">ghc-devs-bounces@haskell.org</a>]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Simon Marlow<br><b>Sent:</b><span class="Apple-converted-space"> </span>22 June 2016 09:51<br><b>To:</b><span class="Apple-converted-space"> </span><a href="mailto:ghc-devs@haskell.org" target="_blank" style="color: purple; text-decoration: underline; ">ghc-devs@haskell.org</a><br><b>Subject:</b><span class="Apple-converted-space"> </span>Require -fexternal-interpreter support for future TH changes?</span><o:p></o:p></div></div></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div><div><div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b>Background</b><o:p></o:p></p></div><p class="MsoNormal" style="margin: 0cm 0cm 6pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">A few months ago I added -fexternal-interpreter to GHC:<o:p></o:p></p><ul type="disc" style="margin-bottom: 0cm; "><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">docs:<span class="Apple-converted-space"> </span><a href="https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d" target="_blank" style="color: purple; text-decoration: underline; ">http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter</a><o:p></o:p></li><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">wiki, rationale:<span class="Apple-converted-space"> </span><a href="https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi" target="_blank" style="color: purple; text-decoration: underline; ">https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi</a><o:p></o:p></li></ul></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.<o:p></o:p></p></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger. It is also now implemented on Windows, thanks to Tamar Christina.<o:p></o:p></p></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b>Proposal</b><o:p></o:p></p></div><div><p class="MsoNormal" style="margin: 0cm 0cm 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.<o:p></o:p></p></div><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Why?<o:p></o:p></div></div><ul type="disc" style="margin-bottom: 0cm; "><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">-fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS.<o:p></o:p></li><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more.<o:p></o:p></li><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.<o:p></o:p></li></ul><div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">How?<o:p></o:p></div></div><div><ul type="disc" style="margin-bottom: 0cm; "><li class="MsoNormal" style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.<o:p></o:p></li></ul><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; "><b>Why now?</b><o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; ">There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter. e.g.<a href="https://phabricator.haskell.org/D2286" target="_blank" style="color: purple; text-decoration: underline; ">https://phabricator.haskell.org/D2286</a><span class="Apple-converted-space"> </span>and<span class="Apple-converted-space"> </span><a href="https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective" target="_blank" style="color: purple; text-decoration: underline; ">https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective</a>, so I'd like to raise it now so we can keep the issue in mind.<o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; ">Cheers<o:p></o:p></p><p style="margin-right: 0cm; margin-left: 0cm; font-size: 12pt; font-family: 'Times New Roman', serif; ">Simon<o:p></o:p></p></div></div></div></div></blockquote></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></blockquote></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div></div></div></div>_______________________________________________<br>ghc-devs mailing list<br><a href="mailto:ghc-devs@haskell.org">ghc-devs@haskell.org</a><br>http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs</div></blockquote></div><br></div></body></html>