The GHC(i)/RTS linker and Template Haskell

[Phyx]

Hi Alexis,

Most information on this can be found on the Wiki,  where a lot of these
design decisions are made. e.g.
https://gitlab.haskell.org/ghc/ghc/-/wikis/dynamic-ghc-programs

The points you've figured out are correct so far, to answer some of your
questions:

> But I don’t actually understand what interpreterDynamic means! The
Haddock comment just says that it determines whether or not the
“interpreter uses the Dynamic way”, but I don’t see why that matters. My
understanding was that GHCi *always* requires dynamic linking, since it is,
after all, loading code dynamically.

DynamicWay essentially means whether or not the runtime linker uses the
platform linker under the hood. When dynamic way your object files will be
linked into a shared library by the RTS linker and that shared library
loaded.
This means that the linker itself doesn't have to do a bunch of work such
as relocation processing etc.  For most Unix platforms this is the default.

The downside of this approach is that on every change, i.e. if you load a
new object file into scope, you have to relink the shared library, unload
the old one, and link the new one in. This brings with it its own set of
problems, such as what happens to references you already hold to symbols on
the old shared library etc.

> Under what circumstances would interpreterDynamic ever be False?

For instance, on Windows. Linking on Windows using the system linker is
generally slower, so creating multiple shared libraries on the fly is time
consuming. There are also some practical issues, for instance base is so
big that it doesn't fit into a single DLL.
or how Windows handles data and code accesses to symbols Shared libraries,
etc.  This means that on Windows we load object files and internally do all
relocation processing, run initializers etc.  Everything you would need to
do to be able to run the code inside the object file.

There are several other platforms as well, such as Android, where there's
no system linker to call etc.

> In the case that interpreterDynamic is True, GHC appears to convert the
desired dyn_o object into a shared library by calling the system linker,
then loads that, which can be very slow but otherwise works. However, when
interpreterDynamic is False, it loads the object directly. Both paths
eventually call into “the RTS linker”, implemented in rts/Linker.c, to
actually load the resulting object.

Yes, the end goal is to be able to resolve a function name to an address.
So whichever strategy is chosen, we must in the end register the functions
with the RTS.  Though loading a shared lib is much less error prone than
loading the object files directly. It also uses less memory and can benefit
from linker level optimizations that we don't implement in the RTS linker.
Also loading a shared library has additional benefits such as that the
system loader deals with running initializers, registering exception
tables, etc.

Hope this clarified it somewhat, but if you have any more questions feel
free to ask.

Regards,
Tamar

On Wed, Jun 1, 2022 at 2:38 AM Alexis King <lexi.lambda at gmail.com> wrote:

> Hi all,
>
> I’ve recently been trying to better understand how and where time is spent
> at compile-time when running Template Haskell splices, and one of the areas
> I’ve been struggling to figure out is the operation of the linker. From
> reading the source code, here’s a summary of what I think I’ve figured out
> so far:
>
>    - TH splices are executed using the GHCi interpreter, though it may be
>    internal or external (if -fexternal-interpreter is used).
>
>    - Regardless of which mode is used, TH splices need their dependencies
>    loaded into the interpreter context before they can be run. This is handled
>    by the call to loadDecls in hscCompileCoreExpr', which in turn calls
>    loadDependencies in GHC.Linker.Loader.
>
>    - loadDependencies loads packages and modules in different ways.
>    Package dependencies are just loaded via the appropriate built shared
>    libraries, but modules from the current package have to be loaded a
>    different way, via loadObjects (also in GHC.Linker.Loader).
>
> Here, however, is where I get a bit lost. GHC has two strategies for
> loading individual objects, which it chooses between depending on whether
> the current value of interpreterDynamic is True. But I don’t actually
> understand what interpreterDynamic means! The Haddock comment just says
> that it determines whether or not the “interpreter uses the Dynamic way”,
> but I don’t see why that matters. My understanding was that GHCi *always*
> requires dynamic linking, since it is, after all, loading code dynamically.
> Under what circumstances would interpreterDynamic ever be False?
>
> Furthermore, I don’t actually understand precisely how and why this
> influences the choice of loading strategy. In the case that
> interpreterDynamic is True, GHC appears to convert the desired dyn_o object
> into a shared library by calling the system linker, then loads that, which
> can be very slow but otherwise works. However, when interpreterDynamic is
> False, it loads the object directly. Both paths eventually call into “the
> RTS linker”, implemented in rts/Linker.c, to actually load the resulting
> object.
>
> I have found precious little information on what the RTS linker does, in
> which contexts it’s used, or how precisely it works. Note
> [runtime-linker-phases] at the top of Linker.c has some information, but
> it’s mostly a high-level description of what the code actually does rather
> than an explanation of its role in the bigger picture. Does anyone know of
> any resources they could possibly point me to that help to explain how all
> the pieces fit together here? I’ve spent quite a bit of time reading the
> code, but I’m afraid I still haven’t managed to see the forest for the
> trees.
>
> Thanks,
> Alexis
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20220601/2c584e41/attachment.html>