Out Of Process Template Haskell

[Simon Peyton Jones]

Luite's "out of process TH" work is amazing.  If someone had suggested it to me I'd have said "it sounds entirely impractical", but he showed that I would have been quite wrong.

I'm open to fixing up GHC, preferably via some plugin variations, to support this.  (I would prefer not to add a whole batch of new dependencies.)

If a little sub-group would like to:
 * Articulate their preferred design (from a user point of view)
   on a wiki page
 * Sketch the implementation plan
 * Build a patch
and get consensus on all of that, let's go for it.  I guess that means at least Luite and Moritz; I'm not sure who else is keen here.

The ball is in your court.

Thank you!

Simon

|  -----Original Message-----
|  From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of
|  Moritz Angermann
|  Sent: 03 December 2014 21:24
|  To: ghc-devs
|  Subject: Out Of Process Template Haskell
|  
|  Hi,
|  
|  as you may or may not know, template haskell is not available to cross
|  compilers.  And therefore cross compiler are devoid of an important
|  feature of haskell.
|  
|    We are currently building ghc cross compiler as stage1 compiler with
|  a foreign target.
|  This means that we use a stage0 compiler (the one that is used to
|  build ghc), to build the ghc that will run on the host and produce
|  binaries for the target (e.g. running on x86_64, building for arm).
|  Template Haskell and GHCi are enabled only in stage2.  To build a
|  stage2 compiler we use the stage1 compiler.  Hence the stage2 compiler
|  runs on the target of the stage1 compiler.  A motivating example could
|  be ghc running on an Intel based Mac OS X targeting iOS (ARM).  A
|  *theoretical* stage2 compiler could have GHCi and Template Haskell,
|  but would run on the iOS Device, which is neat but probably not a
|  practical solution.  Hence a compiler running on a more powerful host,
|  producing executables for a foreign target can be a very good solution
|  when the host is much more powerful. (Ref. [1]). In [2] we can read
|  the following:
|  
|  > Stage 1 does not support interactive execution (GHCi) and Template
|  > Haskell. The reason being that when running byte code we must
|  > dynamically link the packages, and only in stage 2 and later can we
|  > guarantee that the packages we dynamically link are compatible with
|  those that GHC was built against (because they are the very same
|  packages).
|  
|    One of the prominent cross compilers is ghcjs, which admittedly is
|  build a little different than the above cross compiler would be.
|  ghcjs shares the same template haskell issue though.  ghcjs--to my
|  knowledge--followed a few different routes to solve this.  And finally
|  came up with the *out-of-process-template-haskell* solution.
|  
|    The Out Of Process Template Haskell approach works by having a
|  *runner* on the target that waits for the compiler to send the
|  template haskell splices, and compiles those splices on the target,
|  whilte querying the *master* ghc for potential lookups during the
|  splice compilation.  The result is then shipped back to the host to
|  integrate into the produced binary for the target.  Hence allowing the
|  more powerful host to do the bulk of the compilation and using the
|  *runner* as a *slave compiler* on the target.
|  
|  
|       .-- host -.
|  .- target -.
|       |         |  -- ( encounters splice and sends it to runner ) --->
|  |          |
|       |         |
|  |          |
|       |   ghc   | .<---- ( compiles and queries ghc for lookups ) ----.
|  |  runner  |
|       |  master | '----------- ( responds to to queries ) ----------->'
|  |  slave   |
|       |         |
|  |          |
|       |         | <- ( receives the compiled splice from the runner ) -
|  |          |
|       '---------'
|  '----------'
|  
|       Fig. 1: ghc - th runner communication.
|  
|  
|    I have implemented the *out-of-process-template-haskell* approach
|  with the help from luite for ghc as a plugin for stage2 *non*-cross
|  compiler[3] using dynamic libraries and a tcp transport by adapting
|  the code from ghcjs to run on the host.
|  Obviously for a regular stage2 ghc, which has GHCi and Template
|  Haskell, this doesn't really buy anyone anything yet.  Also a plugin
|  is impossible to load into a stage1 compiler yet, as the plugin code
|  is wrapped in #ifdef GHCI sections.
|  
|    SIDENOTE: To allow the out-of-process-template-haskell, the plugin
|  api had to be
|              adapted minimally to allow plugins to install hooks.
|  Patches for 7.8[4]
|              and 7.10[5] are readily available.
|  
|  
|    Clearly the *out-of-process-template-haskell* could be integrated
|  into the ghc tree by hard wiring it in, where the plugin did the
|  wiring ad-hoc previous.  This would imply that ghc would have to
|  integrate all dependencies which *out-of-process-template-haskell*
|  would bring with it.
|  
|    There are now three options discussed on #ghc so far:
|  a) Full integration in ghc, as described above.
|  b) Enabling the plugin interface in stage1 compiler [See Note 1 for
|  details].
|     This is probably only interesting for cross-compiler, where stage1
|  is the final
|     stage on the host. [See Note 2 for an example]
|  c) As proposed by luite: integrate a thin layer in ghc, that uses
|  pipes to communicate
|     with an external program on the same host that ghc is running on.
|  That external
|     program will then communicate with the *runner* on the target.
|  
|    The ultimate goal would be a multi-target compiler, that could
|  produce a stage2 compiler that runs on the host but can produce
|  binaries for multiple targets of which the host it's running on is
|  one.
|  
|    I am in favor of extending the plugin api and making it available to
|  stage1 compiler, because I see much potential in the plugin api.
|  Which--with the noted patches [4][5]-- allows you to install hooks,
|  and hence allow you to hook into the complete compiler pipeline and
|  potential other parts as well.  It also makes developing plugins
|  easier, because they can be developed outside of the ghc source tree.
|  And only be used with cross compilers if we can load plugins in stage1
|  compilers.
|  
|  Cheers,
|   Moritz
|  
|  
|  [1]: https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation
|  [2]:
|  https://ghc.haskell.org/trac/ghc/wiki/Building/Architecture/Idiom/Stag
|  es
|  [3]: https://github.com/angerman/oopth
|  [4]: https://gist.github.com/angerman/7db11c24f8935c73fcf5
|  [5]: https://phabricator.haskell.org/D535
|  Note 1: where the plugin will have to be compiled with a stage0
|  compiler, and may even
|          require a stage0 compiler that is as new as the stage1
|  compiler one wants to build,
|          because the compiler being built has to be binary compatible
|  with the plugin.
|  Note 2: An example would be building 7.10 on the host for the host
|  with stage0=ghc7.8,
|          stage1=ghc7.10(built with ghc7.8), stage2=ghc7.10(built with
|  stage1), *and then*
|          building a ghc cross compiler for arm with stage0'=stage2 and
|  stage1'(ghc7.10 targeting
|          ARM built with ghc7.10 on the host).  And allowing this newly
|  built cross compiler
|          (stage1') to accept -fplugin for plugins built with stage2
|  (the same that built stage1')
|  _______________________________________________
|  ghc-devs mailing list
|  ghc-devs at haskell.org
|  http://www.haskell.org/mailman/listinfo/ghc-devs