Require -fexternal-interpreter support for future TH changes?

[Simon Peyton Jones]

It’s a great start, thanks

Simon

From: Simon Marlow [mailto:marlowsd at gmail.com]
Sent: 22 June 2016 12:32
To: Simon Peyton Jones <simonpj at microsoft.com>
Cc: ghc-devs at haskell.org
Subject: Re: Require -fexternal-interpreter support for future TH changes?

How's this? https://ghc.haskell.org/trac/ghc/wiki/Commentary/Compiler/ExternalInterpreter
I don't want to go into too much detail in the wiki, because details are more likely to stay current if they're in Notes in the code.  There are already a few Notes (e.g. https://phabricator.haskell.org/diffusion/GHC/browse/master/compiler/ghci/GHCi.hs;619958832cbe11096cae3dac9a0a7a5591163a00$86) but if anything is confusing I'll happily add more Notes.
Cheers
Simon

On 22 June 2016 at 11:50, Simon Marlow <marlowsd at gmail.com<mailto:marlowsd at gmail.com>> wrote:
On 22 June 2016 at 11:37, Simon Peyton Jones <simonpj at microsoft.com<mailto:simonpj at microsoft.com>> wrote:
I’m ok with this.   It would certainly be great not to support TWO mechanisms indefinitely.

What are the disadvantages to committing to this path?  Would anyone even notice?

Yes, people who are making changes to TH will need to ensure that their changes work with -fexternal-interpreter.  In some cases that might mean extra work, e.g. if we do TemplateHaskell/Introspective<https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective> then potentially the whole of HsSyn needs to be binary serializable.  That worries me quite a lot - THSyn is big but tractable, Generic deriving handled the generation of the binary instances easily enough, but HsSyn is another matter entirely.

There are a lot of moving parts to the implementation, and I for one am utterly ignorant of how it all works.  I would love to see an implementation overview, either somewhere in the code or on a wiki page.  Things like:

•         How, when, and where in the compiler is the separate process started?

•         How do the compiler and server communicate?  Unix pipes? Is it the same on Windows and Unix?

•         What is serialised, when, and how?  For example, GHC has some TH code to run.  Do we send a syntax tree?  Or compile to bytecode and send that?  Or what?

•         How are external references managed.  E.g. if the code to be run refers to ‘map’ I’m sure we don’t serialise the code for ‘map’.
I’m sure there is a lot more. E.g the [wiki:RemoteGHCi wiki page] refers to “a library implementing a message type…” but I don’t know what that library is called or where it lives.

Yes, we should really have a page in the commentary with an overview of the pieces and the main implementation strategy. I'll write one.
Cheers
Simon


Thanks

Simon

From: ghc-devs [mailto:ghc-devs-bounces at haskell.org<mailto:ghc-devs-bounces at haskell.org>] On Behalf Of Simon Marlow
Sent: 22 June 2016 09:51
To: ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>
Subject: Require -fexternal-interpreter support for future TH changes?

Background
A few months ago I added -fexternal-interpreter to GHC:

  *   docs: http://downloads.haskell.org/~ghc/latest/docs/html/users_guide/ghci.html#ghc-flag--fexternal-interpreter<https://na01.safelinks.protection.outlook.com/?url=http:%2f%2fdownloads.haskell.org%2f~ghc%2flatest%2fdocs%2fhtml%2fusers_guide%2fghci.html%23ghc-flag--fexternal-interpreter&data=01%7C01%7Csimonpj%40064d.mgd.microsoft.com%7C5ca29759b5c9483e03fb08d39a7a60ea%7C72f988bf86f141af91ab2d7cd011db47%7C1&sdata=6t6FA3nUkP5FOi%2fqCBSr3GxMH2rwWNk89je1qBH9GfI%3d>
  *   wiki, rationale: https://ghc.haskell.org/trac/ghc/wiki/RemoteGHCi
When -fexternal-interpreter is used, GHC runs interpreted code in a separate subprocess, and communicates with it using binary messages over a pipe.
-fexternal-interpreter currently implements all of TH, quasi-quoting, annotations, and all the GHCi features except for some features of the debugger.  It is also now implemented on Windows, thanks to Tamar Christina.
Proposal
I'd like to propose that going forward we commit to maintaining full support for -fexternal-interpreter, with a view to making it the default.
Why?

  *   -fexternal-interpreter will be a prerequisite for GHCJS support, so maintaining full support for TH in -fexternal-interpreter will ensure that everything that works with GHC works with GHCJS.
  *   We will be able to make simplifications in GHC and the build system once -fexternal-interpreter is the default, because when compiling with -prof or -dynamic we won't have to compile things twice any more.
  *   Ultimately we don't want to have two ways of doing everything, because that's harder to maintain.
How?

  *   I'll make all the TH and quasi-quoting tests run with and without -fexternal-interpreter, so it will break validate if one of these fails.

Why now?

There are some TH changes in the pipeline that will need special attention to work with -fexternal-interpreter.  e.g. https://phabricator.haskell.org/D2286 and https://ghc.haskell.org/trac/ghc/wiki/TemplateHaskell/Introspective, so I'd like to raise it now so we can keep the issue in mind.



Cheers

Simon


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20160622/d5b58d0d/attachment-0001.html>