Out Of Process Template Haskell

Wed Dec 3 21:24:02 UTC 2014

Hi,

as you may or may not know, template haskell is not available to cross compilers.  And
therefore cross compiler are devoid of an important feature of haskell.

  We are currently building ghc cross compiler as stage1 compiler with a foreign target.
This means that we use a stage0 compiler (the one that is used to build ghc), to build
the ghc that will run on the host and produce binaries for the target (e.g. running on
x86_64, building for arm).  Template Haskell and GHCi are enabled only in stage2.  To
build a stage2 compiler we use the stage1 compiler.  Hence the stage2 compiler runs
on the target of the stage1 compiler.  A motivating example could be ghc running on 
an Intel based Mac OS X targeting iOS (ARM).  A *theoretical* stage2 compiler could have
GHCi and Template Haskell, but would run on the iOS Device, which is neat but probably
not a practical solution.  Hence a compiler running on a more powerful host, producing
executables for a foreign target can be a very good solution when the host is much more
powerful. (Ref. [1]). In [2] we can read the following:

> Stage 1 does not support interactive execution (GHCi) and Template Haskell. The reason
> being that when running byte code we must dynamically link the packages, and only in
> stage 2 and later can we guarantee that the packages we dynamically link are compatible
> with those that GHC was built against (because they are the very same packages).

  One of the prominent cross compilers is ghcjs, which admittedly is build a little 
different than the above cross compiler would be.  ghcjs shares the same template haskell
issue though.  ghcjs--to my knowledge--followed a few different routes to solve this.  And
finally came up with the *out-of-process-template-haskell* solution.

  The Out Of Process Template Haskell approach works by having a *runner* on the target
that waits for the compiler to send the template haskell splices, and compiles those
splices on the target, whilte querying the *master* ghc for potential lookups during the
splice compilation.  The result is then shipped back to the host to integrate into
the produced binary for the target.  Hence allowing the more powerful host to do the
bulk of the compilation and using the *runner* as a *slave compiler* on the target.

     .-- host -.                                                        .- target -.
     |         |  -- ( encounters splice and sends it to runner ) --->  |          |
     |         |                                                        |          |
     |   ghc   | .<---- ( compiles and queries ghc for lookups ) ----.  |  runner  |
     |  master | '----------- ( responds to to queries ) ----------->'  |  slave   |
     |         |                                                        |          |
     |         | <- ( receives the compiled splice from the runner ) -  |          |
     '---------'                                                        '----------'

     Fig. 1: ghc - th runner communication.

  I have implemented the *out-of-process-template-haskell* approach with the help from
luite for ghc as a plugin for stage2 *non*-cross compiler[3] using dynamic libraries and
a tcp transport by adapting the code from ghcjs to run on the host.
Obviously for a regular stage2 ghc, which has GHCi and Template Haskell, this doesn't really
buy anyone anything yet.  Also a plugin is impossible to load into a stage1 compiler yet,
as the plugin code is wrapped in #ifdef GHCI sections.

  SIDENOTE: To allow the out-of-process-template-haskell, the plugin api had to be
            adapted minimally to allow plugins to install hooks.  Patches for 7.8[4]
            and 7.10[5] are readily available.

  Clearly the *out-of-process-template-haskell* could be integrated into the ghc tree
by hard wiring it in, where the plugin did the wiring ad-hoc previous.  This would imply
that ghc would have to integrate all dependencies which *out-of-process-template-haskell*
would bring with it.

  There are now three options discussed on #ghc so far:
a) Full integration in ghc, as described above.
b) Enabling the plugin interface in stage1 compiler [See Note 1 for details].
   This is probably only interesting for cross-compiler, where stage1 is the final
   stage on the host. [See Note 2 for an example]
c) As proposed by luite: integrate a thin layer in ghc, that uses pipes to communicate
   with an external program on the same host that ghc is running on.  That external
   program will then communicate with the *runner* on the target.

  The ultimate goal would be a multi-target compiler, that could produce a stage2 compiler
that runs on the host but can produce binaries for multiple targets of which the host
it's running on is one.

  I am in favor of extending the plugin api and making it available to stage1 compiler,
because I see much potential in the plugin api.  Which--with the noted patches [4][5]--
allows you to install hooks, and hence allow you to hook into the complete compiler
pipeline and potential other parts as well.  It also makes developing plugins easier,
because they can be developed outside of the ghc source tree.  And only be used with
cross compilers if we can load plugins in stage1 compilers.

Cheers,
 Moritz

[1]: https://ghc.haskell.org/trac/ghc/wiki/CrossCompilation
[2]: https://ghc.haskell.org/trac/ghc/wiki/Building/Architecture/Idiom/Stages
[3]: https://github.com/angerman/oopth
[4]: https://gist.github.com/angerman/7db11c24f8935c73fcf5
[5]: https://phabricator.haskell.org/D535
Note 1: where the plugin will have to be compiled with a stage0 compiler, and may even
        require a stage0 compiler that is as new as the stage1 compiler one wants to build,
        because the compiler being built has to be binary compatible with the plugin.
Note 2: An example would be building 7.10 on the host for the host with stage0=ghc7.8,
        stage1=ghc7.10(built with ghc7.8), stage2=ghc7.10(built with stage1), *and then*
        building a ghc cross compiler for arm with stage0'=stage2 and stage1'(ghc7.10 targeting
        ARM built with ghc7.10 on the host).  And allowing this newly built cross compiler
        (stage1') to accept -fplugin for plugins built with stage2 (the same that built stage1')