A macOS static linking mystery

Wed Aug 10 02:09:53 UTC 2022

One peculiarity with the ordering is that linkers only search static
archives for existing undefined references. If the reference to `Cffi`
actually comes first, then nothing should be required from it yet and
*it shouldn't be linked* (absent options like GNU ld's
`--whole-archive`).

This said, ghc also passes a bunch of explicitly undefined symbols
(`-Wl,-u…`) to ld; if those include symbols from `-lCffi` then it will
in fact be linked in.

On Tue, Aug 9, 2022 at 9:59 PM Ryan Scott <ryan.gl.scott at gmail.com> wrote:
>
> I believe I've figured out what is happening here. This will take a bit of explanation, so bear with me.
>
> First, it's important to note that this behavior depends on whether your GHC was configured with the `--with-system-libffi` flag or not. If it was configured with this flag, then GHC will always link against your system's version of libffi, and the behavior seen in this thread would not occur. I am using a bindist provided by ghcup which was _not_ configured with `--with-system-libffi`, so GHC bundles its own copy of libffi. Interestingly, there appear to be at least two copies of libffi that are bundled: one dynamic library, simply named libffi.dylib, and one static library, which is named libCffi.a [1]. Remember the name libCffi.a, since it will come up again later.
>
> Next, let's take a look at how GHC performs the final linking step when compiling an executable. GHC will call the C compiler to perform this final linking step, which will have roughly the following shape on macOS:
>
>     gcc ... -L<ghc-lib-dir>/rts <Haskell-libraries> -lCffi <extra-libraries> -Wl,-dead_strip_dylibs -o Main
>
> I'm omitting quite a few details here, but the important bits are that (1) -lCffi comes before any of the C libraries listed in an `extra-libraries` stanza in a .cabal file, and (2) GHC on macOS will pass -Wl,-dead_strip_dylibs, which causes the linker to remove any references to shared libraries that go unused.
>
> In the particular case of libffi.cabal, we have `extra-libraries: ffi`, so the C compiler is given `-lCffi -lffi` as arguments, in that order. Recall that -libCffi.a (i.e., -lCffi) is a statically linked copy of libffi, whereas libffi.dylib (i.e., -lffi) is a dynamically linked version. The linker will resolve all of its references to libffi functionality from libCffi.a because it appears first, and since the linker never needs to resolve anything from libffi.dylib, the -Wl,-dead_strip_dylibs will remove the reference to libffi.dylib entirely. The result is that the final executable will statically link against libffi. This is why this phenomenon only occurs with libffi and not with other C libraries, since only libffi is given special treatment within GHC like this.
>
> Why, then, does this only happen on macOS and not on, say, Linux? On Linux, the final linking step looks slightly different. Instead of passing -Wl,-dead_strip_dylibs (which is a macOS-specific option), it will pass -Wl,--no-as-needed. Note that -Wl,--no-as-needed has the entirely _opposite_ effect of -Wl,-dead_strip_dylibs: even if a shared library never gets referenced, the linker will still declare a dynamic dependency on it. That is exactly what happens here, as the -lCffi argument will still take precedence over the -lffi argument on Linux, but there will be a useless dynamic dependency on a dynamic libffi library due to -Wl,--no-as-needed.
>
> At this point, I realized that one consequence of all this is that if you want to statically link against against libffi, and you are using a GHC configured without ` --with-system-libffi`, then it suffices to simply compile your program without any `extra-libraries` at all. That is because the final link step will pass `-lCffi` no matter what, so the resulting executable will always statically link against libffi by way of libCffi.a. Moreover, libCffi is provided on all platforms, not just macOS, so this trick is reasonably portable.
>
> -----
>
> Having explained all this, I am forced to wonder if this is intended behavior of GHC, or if it is a series of coincidences. The fact that macOS and Linux pass -Wl,-dead_strip_dylibs and -Wl,--no-as-needed, two flags with completely opposite behavior, is certainly a little unusual. But these choices appear to be deliberate, as they are explained in Notes in the GHC source code here [2] (for macOS) and here [3] (for Linux). The other possibly unusual thing is that GHC always passes -lCffi before any of the `extra-libraries`. That could very well be just a coincidence—I'm not sure if there's a particular reason for -lCffi to come first.
>
> The reason that I'm looking so deeply into this matter is that I'm trying to see what it would take to reliably link against libffi statically in the Haskell bindings [4]. I've discovered a trick above to make it work for most versions of GHC: just rely on `-lCffi`. Does this trick seem like it would continue to work for the foreseeable future, or is this approach liable to break in a future release?
>
> Ryan
> -----
> [1] See https://gitlab.haskell.org/ghc/ghc/-/issues/20395 for more on this unusual amount of library duplication.
> [2] https://gitlab.haskell.org/ghc/ghc/-/blob/d71a20514546e0befe6e238d0658cbaad5a13996/compiler/GHC/Driver/Pipeline.hs#L293-340
> [3] https://gitlab.haskell.org/ghc/ghc/-/blob/d71a20514546e0befe6e238d0658cbaad5a13996/compiler/GHC/SysTools/Info.hs#L57-69
> [4] https://github.com/remiturk/libffi/issues/6
>
> On Tue, Aug 9, 2022 at 9:58 AM Ryan Scott <ryan.gl.scott at gmail.com> wrote:
>>
>> No, I'm using macOS 10.15.7 (BuildVersion 19H2) on an x86_64 machine.
>>
>> Ryan

-- 
brandon s allbery kf8nh
allbery.b at gmail.com