Require -fexternal-interpreter support for future TH changes?

Wed Jun 22 14:38:47 UTC 2016

Thanks for the feedback Richard.  It's helpful to know which parts of the
implementation you found non-obvious.  I will write some more Notes, but
first:

* On why we need a RemoteRef of an IORef - because the IORef lives on the
server, so GHC has a RemoteRef to it.   Did you see Note [External GHCi
pointers] in compiler/ghci/GHCi.hs?  Does that help?  Should I xref it from
somewhere else?

* We probably could separate Message into two types, it's a good idea.
I'll look into doing that.

Cheers
Simon

On 22 June 2016 at 14:36, Richard Eisenberg <eir at cis.upenn.edu> wrote:

> It seems there are nice advantages to this approach, and so I'm fine
> requiring support from TH. But here is a comment I just posted on D2286
> requesting more comments in the code:
>
> ---
> I've just spent some time staring at GHCi/TH.hs and friends, and it's
> starting to make sense. Would it be possible to add some comments/Notes
> there? It would be great to know what is run on the client (GHC, right?)
> and what on the server (TH, right?). Also, why do we need a RemoteRef of
> an IORef? This, at first, strikes me as odd. I'm sure it makes great
> sense once I understand all the other moving parts, but it would be helpful
> to have a primer. I also find it a bit confusing that the Message type
> contains both messages to the server and messages from the server. For
> example, I would expect StartTH and Reify to live in different datatypes,
> because the two should never mix. Or am I horribly mistaken?
> ---
>
> Before we require -fexternal-interpreter (which seems to be the endgoal
> here), has anyone looked at performance implications? All the marshaling
> and unmarshaling has to take a toll.
>
> Richard
>
> On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs <
> ghc-devs at haskell.org> wrote:
>
> It’s a great start, thanks
>
> Simon
>
> *From:* Simon Marlow [mailto:marlowsd at gmail.com]
> *Sent:* 22 June 2016 12:32
> *To:* Simon Peyton Jones <simonpj at microsoft.com>
> *Cc:* ghc-devs at haskell.org
> *Subject:* Re: Require -fexternal-interpreter support for future TH
> changes?
>
>
> How's this?
> https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
>
> I don't want to go into too much detail in the wiki, because details are
> more likely to stay current if they're in Notes in the code.  There are
> already a few Notes (e.g.
> https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
> but if anything is confusing I'll happily add more Notes.
>
> Cheers
>
> Simon
>
>
>
> On 22 June 2016 at 11:50, Simon Marlow <marlowsd at gmail.com> wrote:
>
> On 22 June 2016 at 11:37, Simon Peyton Jones <simonpj at microsoft.com>
> wrote:
>
> I’m ok with this.   *It would certainly be great not to support TWO
> mechanisms indefinitely*.
>
> What are the disadvantages to committing to this path?  Would anyone even
> notice?
>
>
> Yes, people who are making changes to TH will need to ensure that their
> changes work with -fexternal-interpreter.  In some cases that might mean
> extra work, e.g. if we do TemplateHaskell/Introspective
> <https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective> then
> potentially the whole of HsSyn needs to be binary serializable.  That
> worries me quite a lot - THSyn is big but tractable, Generic deriving
> handled the generation of the binary instances easily enough, but HsSyn is
> another matter entirely.
>
>
> There are a lot of moving parts to the implementation, and I for one am
> utterly ignorant of how it all works.  I would love to see an
> implementation overview, either somewhere in the code or on a wiki page.
> Things like:
>
> ·         How, when, and where in the compiler is the separate process
> started?
>
> ·         How do the compiler and server communicate?  Unix pipes? Is it
> the same on Windows and Unix?
>
> ·         What is serialised, when, and how?  For example, GHC has some
> TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send
> that?  Or what?
>
> ·         How are external references managed.  E.g. if the code to be
> run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers
> to “a library implementing a message type…” but I don’t know what that
> library is called or where it lives.
>
>
>
> Yes, we should really have a page in the commentary with an overview of
> the pieces and the main implementation strategy. I'll write one.
> Cheers
> Simon
>
>
>
> Thanks
>
> Simon
>
> *From:* ghc-devs [mailto:ghc-devs-bounces at haskell.org] *On Behalf Of *Simon
> Marlow
> *Sent:* 22 June 2016 09:51
> *To:* ghc-devs at haskell.org
> *Subject:* Require -fexternal-interpreter support for future TH changes?
>
>
> *Background*
>
> A few months ago I added -fexternal-interpreter to GHC:
>
>    - docs:
>    http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
>    <https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
>    - wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
>
> When -fexternal-interpreter is used, GHC runs interpreted code in a
> separate subprocess, and communicates with it using binary messages over a
> pipe.
>
> -fexternal-interpreter currently implements all of TH, quasi-quoting,
> annotations, and all the GHCi features except for some features of the
> debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
>
> *Proposal*
>
> I'd like to propose that going forward we commit to maintaining full
> support for -fexternal-interpreter, with a view to making it the default.
> Why?
>
>    - -fexternal-interpreter will be a prerequisite for GHCJS support, so
>    maintaining full support for TH in -fexternal-interpreter will ensure that
>    everything that works with GHC works with GHCJS.
>    - We will be able to make simplifications in GHC and the build system
>    once -fexternal-interpreter is the default, because when compiling with
>    -prof or -dynamic we won't have to compile things twice any more.
>    - Ultimately we don't want to have two ways of doing everything,
>    because that's harder to maintain.
>
> How?
>
>    - I'll make all the TH and quasi-quoting tests run with and without
>    -fexternal-interpreter, so it will break validate if one of these fails.
>
> *Why now?*
>
> There are some TH changes in the pipeline that will need special attention
> to work with -fexternal-interpreter.  e.g.
> https://phabricator.haskell.org/D2286 and
> https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so
> I'd like to raise it now so we can keep the issue in mind.
>
>
>
> Cheers
>
> Simon
>
>
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20160622/5a1fdc4f/attachment-0001.html>