Require -fexternal-interpreter support for future TH changes?

Wed Jun 22 15:02:10 UTC 2016

One other thing:

On 22 June 2016 at 14:36, Richard Eisenberg <eir at cis.upenn.edu> wrote:

> It seems there are nice advantages to this approach, and so I'm fine
> requiring support from TH. But here is a comment I just posted on D2286
> requesting more comments in the code:
>
> ---
> I've just spent some time staring at GHCi/TH.hs and friends, and it's
> starting to make sense. Would it be possible to add some comments/Notes
> there? It would be great to know what is run on the client (GHC, right?)
> and what on the server (TH, right?). Also, why do we need a RemoteRef of
> an IORef? This, at first, strikes me as odd. I'm sure it makes great
> sense once I understand all the other moving parts, but it would be helpful
> to have a primer. I also find it a bit confusing that the Message type
> contains both messages to the server and messages from the server. For
> example, I would expect StartTH and Reify to live in different datatypes,
> because the two should never mix. Or am I horribly mistaken?
> ---
>
> Before we require -fexternal-interpreter (which seems to be the endgoal
> here), has anyone looked at performance implications? All the marshaling
> and unmarshaling has to take a toll.
>

Yes, I've looked at performance and done a fair bit of optimisation of the
current implementation, but mainly for GHCi, not so much for TH yet.  For
GHCi I think it was <10% of compile time, mainly in shipping the byte code
over the pipe.  I'll need to do some measurements of TH performance before
making it the default.  We can probably claw back quite a lot of perf by
switching from binary to something else (binary-cbor or store).

Cheers
Simon

>
> Richard
>
> On Jun 22, 2016, at 8:27 AM, Simon Peyton Jones via ghc-devs <
> ghc-devs at haskell.org> wrote:
>
> It’s a great start, thanks
>
> Simon
>
> *From:* Simon Marlow [mailto:marlowsd at gmail.com]
> *Sent:* 22 June 2016 12:32
> *To:* Simon Peyton Jones <simonpj at microsoft.com>
> *Cc:* ghc-devs at haskell.org
> *Subject:* Re: Require -fexternal-interpreter support for future TH
> changes?
>
>
> How's this?
> https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
>
> I don't want to go into too much detail in the wiki, because details are
> more likely to stay current if they're in Notes in the code.  There are
> already a few Notes (e.g.
> https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86)
> but if anything is confusing I'll happily add more Notes.
>
> Cheers
>
> Simon
>
>
>
> On 22 June 2016 at 11:50, Simon Marlow <marlowsd at gmail.com> wrote:
>
> On 22 June 2016 at 11:37, Simon Peyton Jones <simonpj at microsoft.com>
> wrote:
>
> I’m ok with this.   *It would certainly be great not to support TWO
> mechanisms indefinitely*.
>
> What are the disadvantages to committing to this path?  Would anyone even
> notice?
>
>
> Yes, people who are making changes to TH will need to ensure that their
> changes work with -fexternal-interpreter.  In some cases that might mean
> extra work, e.g. if we do TemplateHaskell/Introspective
> <https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective> then
> potentially the whole of HsSyn needs to be binary serializable.  That
> worries me quite a lot - THSyn is big but tractable, Generic deriving
> handled the generation of the binary instances easily enough, but HsSyn is
> another matter entirely.
>
>
> There are a lot of moving parts to the implementation, and I for one am
> utterly ignorant of how it all works.  I would love to see an
> implementation overview, either somewhere in the code or on a wiki page.
> Things like:
>
> ·         How, when, and where in the compiler is the separate process
> started?
>
> ·         How do the compiler and server communicate?  Unix pipes? Is it
> the same on Windows and Unix?
>
> ·         What is serialised, when, and how?  For example, GHC has some
> TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send
> that?  Or what?
>
> ·         How are external references managed.  E.g. if the code to be
> run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
> I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers
> to “a library implementing a message type…” but I don’t know what that
> library is called or where it lives.
>
>
>
> Yes, we should really have a page in the commentary with an overview of
> the pieces and the main implementation strategy. I'll write one.
> Cheers
> Simon
>
>
>
> Thanks
>
> Simon
>
> *From:* ghc-devs [mailto:ghc-devs-bounces at haskell.org] *On Behalf Of *Simon
> Marlow
> *Sent:* 22 June 2016 09:51
> *To:* ghc-devs at haskell.org
> *Subject:* Require -fexternal-interpreter support for future TH changes?
>
>
> *Background*
>
> A few months ago I added -fexternal-interpreter to GHC:
>
>    - docs:
>    http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter
>    <https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
>    - wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
>
> When -fexternal-interpreter is used, GHC runs interpreted code in a
> separate subprocess, and communicates with it using binary messages over a
> pipe.
>
> -fexternal-interpreter currently implements all of TH, quasi-quoting,
> annotations, and all the GHCi features except for some features of the
> debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
>
> *Proposal*
>
> I'd like to propose that going forward we commit to maintaining full
> support for -fexternal-interpreter, with a view to making it the default.
> Why?
>
>    - -fexternal-interpreter will be a prerequisite for GHCJS support, so
>    maintaining full support for TH in -fexternal-interpreter will ensure that
>    everything that works with GHC works with GHCJS.
>    - We will be able to make simplifications in GHC and the build system
>    once -fexternal-interpreter is the default, because when compiling with
>    -prof or -dynamic we won't have to compile things twice any more.
>    - Ultimately we don't want to have two ways of doing everything,
>    because that's harder to maintain.
>
> How?
>
>    - I'll make all the TH and quasi-quoting tests run with and without
>    -fexternal-interpreter, so it will break validate if one of these fails.
>
> *Why now?*
>
> There are some TH changes in the pipeline that will need special attention
> to work with -fexternal-interpreter.  e.g.
> https://phabricator.haskell.org/D2286 and
> https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so
> I'd like to raise it now so we can keep the issue in mind.
>
>
>
> Cheers
>
> Simon
>
>
>
>
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20160622/53016d27/attachment-0001.html>