Require -fexternal-interpreter support for future TH changes?

Wed Jun 22 13:36:03 UTC 2016

It seems there are nice advantages to this approach, and so I'm fine requiring support from TH. But here is a comment I just posted on D2286 requesting more comments in the code:

---
I've just spent some time staring at GHCi/TH.hs and friends, and it's starting to make sense. Would it be possible to add some comments/Notes there? It would be great to know what is run on the client (GHC, right?) and what on the server (TH, right?). Also, why do we need a RemoteRef of an IORef? This, at first, strikes me as odd. I'm sure it makes great sense once I understand all the other moving parts, but it would be helpful to have a primer. I also find it a bit confusing that the Message type contains both messages to the server and messages from the server. For example, I would expect StartTH and Reify to live in different datatypes, because the two should never mix. Or am I horribly mistaken?
---

Before we require -fexternal-interpreter (which seems to be the endgoal here), has anyone looked at performance implications? All the marshaling and unmarshaling has to take a toll.

Richard

On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs <ghc-devs at haskell.org> wrote:

> It’s a great start, thanks
>  
> Simon
>  
> From: Simon Marlow [mailto:marlowsd at gmail.com] 
> Sent: 22 June 2016 12:32
> To: Simon Peyton Jones <simonpj at microsoft.com>
> Cc: ghc-devs at haskell.org
> Subject: Re: Require -fexternal-interpreter support for future TH changes?
>  
> How's this? https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
> 
> I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code.  There are already a few Notes (e.g.https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86) but if anything is confusing I'll happily add more Notes.
> 
> Cheers
> 
> Simon
> 
>  
> 
> On 22 June 2016 at 11:50, Simon Marlow <marlowsd at gmail.com> wrote:
> 
> On 22 June 2016 at 11:37, Simon Peyton Jones <simonpj at microsoft.com> wrote:
> 
> I’m ok with this.   It would certainly be great not to support TWO mechanisms indefinitely.
>  
> What are the disadvantages to committing to this path?  Would anyone even notice?
>  
> Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter.  In some cases that might mean extra work, e.g. if we do TemplateHaskell/Introspective then potentially the whole of HsSyn needs to be binary serializable.  That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.
>  
> There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works.  I would love to see an implementation overview, either somewhere in the code or on a wiki page.  Things like:
> ·         How, when, and where in the compiler is the separate process started?
> 
> ·         How do the compiler and server communicate?  Unix pipes? Is it the same on Windows and Unix?
> 
> ·         What is serialised, when, and how?  For example, GHC has some TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send that?  Or what?
> 
> ·         How are external references managed.  E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
> 
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.
>  
> Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.
> 
> Cheers
> Simon
>  
>  
> Thanks
>  
> Simon
>  
> From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of Simon Marlow
> Sent: 22 June 2016 09:51
> To: ghc-devs at haskell.org
> Subject: Require -fexternal-interpreter support for future TH changes?
>  
> Background
> 
> A few months ago I added -fexternal-interpreter to GHC:
> 
> docs: http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
> wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
> When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.
> 
> -fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
> 
> Proposal
> 
> I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.
> 
> Why?
> -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS.
> We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more.
> Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.
> How?
> I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.
> Why now?
> 
> There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter.  e.g.https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.
> 
>  
> 
> Cheers
> 
> Simon
> 
>  
>  
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20160622/de84867e/attachment-0001.html>