A macOS static linking mystery

[Ryan Scott]

I believe I've figured out what is happening here. This will take a bit of
explanation, so bear with me.

First, it's important to note that this behavior depends on whether your
GHC was configured with the `--with-system-libffi` flag or not. If it was
configured with this flag, then GHC will always link against your system's
version of libffi, and the behavior seen in this thread would not occur. I
am using a bindist provided by ghcup which was _not_ configured with
`--with-system-libffi`, so GHC bundles its own copy of libffi.
Interestingly, there appear to be at least two copies of libffi that are
bundled: one dynamic library, simply named libffi.dylib, and one static
library, which is named libCffi.a [1]. Remember the name libCffi.a, since
it will come up again later.

Next, let's take a look at how GHC performs the final linking step when
compiling an executable. GHC will call the C compiler to perform this final
linking step, which will have roughly the following shape on macOS:

    gcc ... -L<ghc-lib-dir>/rts <Haskell-libraries> -lCffi
<extra-libraries> -Wl,-dead_strip_dylibs -o Main

I'm omitting quite a few details here, but the important bits are that (1)
-lCffi comes before any of the C libraries listed in an `extra-libraries`
stanza in a .cabal file, and (2) GHC on macOS will pass
-Wl,-dead_strip_dylibs, which causes the linker to remove any references to
shared libraries that go unused.

In the particular case of libffi.cabal, we have `extra-libraries: ffi`, so
the C compiler is given `-lCffi -lffi` as arguments, in that order. Recall
that -libCffi.a (i.e., -lCffi) is a statically linked copy of libffi,
whereas libffi.dylib (i.e., -lffi) is a dynamically linked version. The
linker will resolve all of its references to libffi functionality from
libCffi.a because it appears first, and since the linker never needs to
resolve anything from libffi.dylib, the -Wl,-dead_strip_dylibs will remove
the reference to libffi.dylib entirely. The result is that the final
executable will statically link against libffi. This is why this phenomenon
only occurs with libffi and not with other C libraries, since only libffi
is given special treatment within GHC like this.

Why, then, does this only happen on macOS and not on, say, Linux? On Linux,
the final linking step looks slightly different. Instead of passing
-Wl,-dead_strip_dylibs (which is a macOS-specific option), it will pass
-Wl,--no-as-needed. Note that -Wl,--no-as-needed has the entirely
_opposite_ effect of -Wl,-dead_strip_dylibs: even if a shared library never
gets referenced, the linker will still declare a dynamic dependency on it.
That is exactly what happens here, as the -lCffi argument will still take
precedence over the -lffi argument on Linux, but there will be a useless
dynamic dependency on a dynamic libffi library due to -Wl,--no-as-needed.

At this point, I realized that one consequence of all this is that if you
want to statically link against against libffi, and you are using a GHC
configured without ` --with-system-libffi`, then it suffices to simply
compile your program without any `extra-libraries` at all. That is because
the final link step will pass `-lCffi` no matter what, so the resulting
executable will always statically link against libffi by way of libCffi.a.
Moreover, libCffi is provided on all platforms, not just macOS, so this
trick is reasonably portable.

-----

Having explained all this, I am forced to wonder if this is intended
behavior of GHC, or if it is a series of coincidences. The fact that macOS
and Linux pass -Wl,-dead_strip_dylibs and -Wl,--no-as-needed, two flags
with completely opposite behavior, is certainly a little unusual. But these
choices appear to be deliberate, as they are explained in Notes in the GHC
source code here [2] (for macOS) and here [3] (for Linux). The other
possibly unusual thing is that GHC always passes -lCffi before any of the
`extra-libraries`. That could very well be just a coincidence—I'm not sure
if there's a particular reason for -lCffi to come first.

The reason that I'm looking so deeply into this matter is that I'm trying
to see what it would take to reliably link against libffi statically in the
Haskell bindings [4]. I've discovered a trick above to make it work for
most versions of GHC: just rely on `-lCffi`. Does this trick seem like it
would continue to work for the foreseeable future, or is this approach
liable to break in a future release?

Ryan
-----
[1] See https://gitlab.haskell.org/ghc/ghc/-/issues/20395 for more on this
unusual amount of library duplication.
[2]
https://gitlab.haskell.org/ghc/ghc/-/blob/d71a20514546e0befe6e238d0658cbaad5a13996/compiler/GHC/Driver/Pipeline.hs#L293-340
[3]
https://gitlab.haskell.org/ghc/ghc/-/blob/d71a20514546e0befe6e238d0658cbaad5a13996/compiler/GHC/SysTools/Info.hs#L57-69
[4] https://github.com/remiturk/libffi/issues/6

On Tue, Aug 9, 2022 at 9:58 AM Ryan Scott <ryan.gl.scott at gmail.com> wrote:

> No, I'm using macOS 10.15.7 (BuildVersion 19H2) on an x86_64 machine.
>
> Ryan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20220809/cc520535/attachment.html>