Out Of Process Template Haskell
Simon Peyton Jones
simonpj at microsoft.com
Thu Dec 4 11:07:48 UTC 2014
Luite's "out of process TH" work is amazing. If someone had suggested it to me I'd have said "it sounds entirely impractical", but he showed that I would have been quite wrong.
I'm open to fixing up GHC, preferably via some plugin variations, to support this. (I would prefer not to add a whole batch of new dependencies.)
If a little sub-group would like to:
* Articulate their preferred design (from a user point of view)
on a wiki page
* Sketch the implementation plan
* Build a patch
and get consensus on all of that, let's go for it. I guess that means at least Luite and Moritz; I'm not sure who else is keen here.
The ball is in your court.
| -----Original Message-----
| From: ghc-devs [mailto:ghc-devs-bounces at haskell.org] On Behalf Of
| Moritz Angermann
| Sent: 03 December 2014 21:24
| To: ghc-devs
| Subject: Out Of Process Template Haskell
| as you may or may not know, template haskell is not available to cross
| compilers. And therefore cross compiler are devoid of an important
| feature of haskell.
| We are currently building ghc cross compiler as stage1 compiler with
| a foreign target.
| This means that we use a stage0 compiler (the one that is used to
| build ghc), to build the ghc that will run on the host and produce
| binaries for the target (e.g. running on x86_64, building for arm).
| Template Haskell and GHCi are enabled only in stage2. To build a
| stage2 compiler we use the stage1 compiler. Hence the stage2 compiler
| runs on the target of the stage1 compiler. A motivating example could
| be ghc running on an Intel based Mac OS X targeting iOS (ARM). A
| *theoretical* stage2 compiler could have GHCi and Template Haskell,
| but would run on the iOS Device, which is neat but probably not a
| practical solution. Hence a compiler running on a more powerful host,
| producing executables for a foreign target can be a very good solution
| when the host is much more powerful. (Ref. ). In  we can read
| the following:
| > Stage 1 does not support interactive execution (GHCi) and Template
| > Haskell. The reason being that when running byte code we must
| > dynamically link the packages, and only in stage 2 and later can we
| > guarantee that the packages we dynamically link are compatible with
| those that GHC was built against (because they are the very same
| One of the prominent cross compilers is ghcjs, which admittedly is
| build a little different than the above cross compiler would be.
| ghcjs shares the same template haskell issue though. ghcjs--to my
| knowledge--followed a few different routes to solve this. And finally
| came up with the *out-of-process-template-haskell* solution.
| The Out Of Process Template Haskell approach works by having a
| *runner* on the target that waits for the compiler to send the
| template haskell splices, and compiles those splices on the target,
| whilte querying the *master* ghc for potential lookups during the
| splice compilation. The result is then shipped back to the host to
| integrate into the produced binary for the target. Hence allowing the
| more powerful host to do the bulk of the compilation and using the
| *runner* as a *slave compiler* on the target.
| .-- host -.
| .- target -.
| | | -- ( encounters splice and sends it to runner ) --->
| | |
| | |
| | |
| | ghc | .<---- ( compiles and queries ghc for lookups ) ----.
| | runner |
| | master | '----------- ( responds to to queries ) ----------->'
| | slave |
| | |
| | |
| | | <- ( receives the compiled splice from the runner ) -
| | |
| Fig. 1: ghc - th runner communication.
| I have implemented the *out-of-process-template-haskell* approach
| with the help from luite for ghc as a plugin for stage2 *non*-cross
| compiler using dynamic libraries and a tcp transport by adapting
| the code from ghcjs to run on the host.
| Obviously for a regular stage2 ghc, which has GHCi and Template
| Haskell, this doesn't really buy anyone anything yet. Also a plugin
| is impossible to load into a stage1 compiler yet, as the plugin code
| is wrapped in #ifdef GHCI sections.
| SIDENOTE: To allow the out-of-process-template-haskell, the plugin
| api had to be
| adapted minimally to allow plugins to install hooks.
| Patches for 7.8
| and 7.10 are readily available.
| Clearly the *out-of-process-template-haskell* could be integrated
| into the ghc tree by hard wiring it in, where the plugin did the
| wiring ad-hoc previous. This would imply that ghc would have to
| integrate all dependencies which *out-of-process-template-haskell*
| would bring with it.
| There are now three options discussed on #ghc so far:
| a) Full integration in ghc, as described above.
| b) Enabling the plugin interface in stage1 compiler [See Note 1 for
| This is probably only interesting for cross-compiler, where stage1
| is the final
| stage on the host. [See Note 2 for an example]
| c) As proposed by luite: integrate a thin layer in ghc, that uses
| pipes to communicate
| with an external program on the same host that ghc is running on.
| That external
| program will then communicate with the *runner* on the target.
| The ultimate goal would be a multi-target compiler, that could
| produce a stage2 compiler that runs on the host but can produce
| binaries for multiple targets of which the host it's running on is
| I am in favor of extending the plugin api and making it available to
| stage1 compiler, because I see much potential in the plugin api.
| Which--with the noted patches -- allows you to install hooks,
| and hence allow you to hook into the complete compiler pipeline and
| potential other parts as well. It also makes developing plugins
| easier, because they can be developed outside of the ghc source tree.
| And only be used with cross compilers if we can load plugins in stage1
| : https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation
| : https://github.com/angerman/oopth
| : https://gist.github.com/angerman/7db11c24f8935c73fcf5
| : https://phabricator.haskell.org/D535
| Note 1: where the plugin will have to be compiled with a stage0
| compiler, and may even
| require a stage0 compiler that is as new as the stage1
| compiler one wants to build,
| because the compiler being built has to be binary compatible
| with the plugin.
| Note 2: An example would be building 7.10 on the host for the host
| with stage0=ghc7.8,
| stage1=ghc7.10(built with ghc7.8), stage2=ghc7.10(built with
| stage1), *and then*
| building a ghc cross compiler for arm with stage0'=stage2 and
| stage1'(ghc7.10 targeting
| ARM built with ghc7.10 on the host). And allowing this newly
| built cross compiler
| (stage1') to accept -fplugin for plugins built with stage2
| (the same that built stage1')
| ghc-devs mailing list
| ghc-devs at haskell.org
More information about the ghc-devs