[Git][ghc/ghc][wip/angerman/ghc-8.10-backport-rpath] 5 commits: Bump text submodule to 1.2.4.0+
Ben Gamari
gitlab at gitlab.haskell.org
Wed Oct 14 17:51:39 UTC 2020
Ben Gamari pushed to branch wip/angerman/ghc-8.10-backport-rpath at Glasgow Haskell Compiler / GHC
Commits:
a259e6da by Ben Gamari at 2020-10-06T11:00:27-04:00
Bump text submodule to 1.2.4.0+
Fixes #18588 and #17956.
- - - - -
8a85216c by Ömer Sinan Ağacan at 2020-10-06T11:00:27-04:00
Fix uninitialized field read in Linker.c
Valgrind report of the bug when running the test `linker_unload`:
==29666== Conditional jump or move depends on uninitialised value(s)
==29666== at 0x369C5B4: setOcInitialStatus (Linker.c:1305)
==29666== by 0x369C6C5: mkOc (Linker.c:1347)
==29666== by 0x36C027A: loadArchive_ (LoadArchive.c:522)
==29666== by 0x36C0600: loadArchive (LoadArchive.c:626)
==29666== by 0x2C144CD: ??? (in /home/omer/haskell/ghc_2/testsuite/tests/rts/linker/linker_unload.run/linker_unload)
==29666==
==29666== Conditional jump or move depends on uninitialised value(s)
==29666== at 0x369C5B4: setOcInitialStatus (Linker.c:1305)
==29666== by 0x369C6C5: mkOc (Linker.c:1347)
==29666== by 0x369C9F6: preloadObjectFile (Linker.c:1507)
==29666== by 0x369CA8D: loadObj_ (Linker.c:1536)
==29666== by 0x369CB17: loadObj (Linker.c:1557)
==29666== by 0x3866BC: main (linker_unload.c:33)
The problem is `mkOc` allocates a new `ObjectCode` and calls
`setOcInitialStatus` without initializing the `status` field.
`setOcInitialStatus` reads the field as first thing:
static void setOcInitialStatus(ObjectCode* oc) {
if (oc->status == OBJECT_DONT_RESOLVE)
return;
if (oc->archiveMemberName == NULL) {
oc->status = OBJECT_NEEDED;
} else {
oc->status = OBJECT_LOADED;
}
}
`setOcInitialStatus` is unsed in two places for two different purposes:
in `mkOc` where we don't have the `status` field initialized yet (`mkOc`
is supposed to initialize it), and `loadOc` where we do have `status`
field initialized and we want to update it. Instead of splitting the
function into two functions which are both called just once I inline the
functions in the use sites and remove it.
Fixes #18342
(cherry picked from commit 08c1cb0f30770acbf366423f085f8ef92f7f6a06)
- - - - -
fec3a3cf by GHC GitLab CI at 2020-10-06T11:00:27-04:00
configure: Avoid hard-coded ld path on Windows
The fix to #17962 ended up regressing on Windows as it failed to
replicate the logic responsible for overriding the toolchain paths on
Windows. This resulted in a hard-coded path to a directory that likely
doesn't exist on the user's system (#18550).
(cherry picked from commit 34e0fa963f35a77093fc7111a80c557fc6bd614f)
- - - - -
a491d1cb by Ben Gamari at 2020-10-08T08:51:10-04:00
Fix Windows build
- - - - -
43f97049 by Moritz Angermann at 2020-10-14T13:51:31-04:00
[macOS] improved runpath handling
In b592bd98ff25730bbe3c13d6f62a427df8c78e28 we started using
-dead_strip_dylib on macOS when lining dynamic libraries and binaries.
The underlying reason being the Load Command Size Limit in macOS
Sierra (10.14) and later.
GHC will produce @rpath/libHS... dependency entries together with a
corresponding RPATH entry pointing to the location of the libHS...
library. Thus for every library we produce two Load Commands. One to
specify the dependent library, and one with the path where to find it.
This makes relocating libraries and binaries easier, as we just need to
update the RPATH entry with the install_name_tool. The dynamic linker
will then subsitute each @rpath with the RPATH entries it finds in the
libraries load commands or the environement, when looking up @rpath
relative libraries.
-dead_strip_dylibs intructs the linker to drop unused libraries. This in
turn help us reduce the number of referenced libraries, and subsequently
the size of the load commands. This however does not remove the RPATH
entries. Subsequently we can end up (in extreme cases) with only a
single @rpath/libHS... entry, but 100s or more RPATH entries in the Load
Commands.
This patch rectifies this (slighly unorthodox) by passing *no* -rpath
arguments to the linker at link time, but -headerpad 8000. The
headerpad argument is in hexadecimal and the maxium 32k of the load
command size. This tells the linker to pad the load command section
enough for us to inject the RPATHs later. We then proceed to link the
library or binary with -dead_strip_dylibs, and *after* the linking
inspect the library to find the left over (non-dead-stripped)
dependencies (using otool). We find the corresponding RPATHs for each
@rpath relative dependency, and inject them into the library or binary
using the install_name_tool. Thus achieving a deadstripped dylib (and
rpaths) build product.
We can not do this in GHC, without starting to reimplement a dynamic
linker as we do not know which symbols and subsequently libraries are
necessary.
Commissioned-by: Mercury Technologies, Inc. (mercury.com)
(cherry picked from commit 4ff93292243888545da452ea4d4c1987f2343591)
Signed-off-by: Moritz Angermann <moritz.angermann at iohk.io>
- - - - -
23 changed files:
- aclocal.m4
- compiler/ghci/Linker.hs
- compiler/main/DriverPipeline.hs
- compiler/main/DynFlags.hs
- compiler/main/Settings.hs
- compiler/main/SysTools.hs
- compiler/main/SysTools/Settings.hs
- compiler/main/SysTools/Tasks.hs
- compiler/main/ToolSettings.hs
- configure.ac
- docs/users_guide/phases.rst
- ghc.mk
- hadrian/cfg/system.config.in
- hadrian/src/Builder.hs
- hadrian/src/Oracles/Setting.hs
- hadrian/src/Rules/Generate.hs
- includes/ghc.mk
- libraries/text
- mk/config.mk.in
- rts/Linker.c
- rules/build-package-way.mk
- testsuite/tests/rts/all.T
- utils/ghc-cabal/ghc.mk
Changes:
=====================================
aclocal.m4
=====================================
@@ -516,6 +516,10 @@ AC_DEFUN([FP_SETTINGS],
SettingsHaskellCPPCommand="${mingw_bin_prefix}gcc.exe"
SettingsHaskellCPPFlags="$HaskellCPPArgs"
SettingsLdCommand="${mingw_bin_prefix}ld.exe"
+ # Overrides FIND_MERGE_OBJECTS in order to avoid hard-coding linker
+ # path on Windows (#18550).
+ SettingsMergeObjectsCommand="${SettingsLdCommand}"
+ SettingsMergeObjectsFlags="-r --oformat=pe-bigobj-x86-64"
SettingsArCommand="${mingw_bin_prefix}ar.exe"
SettingsRanlibCommand="${mingw_bin_prefix}ranlib.exe"
SettingsDllWrapCommand="${mingw_bin_prefix}dllwrap.exe"
@@ -529,6 +533,8 @@ AC_DEFUN([FP_SETTINGS],
SettingsHaskellCPPCommand="$(basename $HaskellCPPCmd)"
SettingsHaskellCPPFlags="$HaskellCPPArgs"
SettingsLdCommand="$(basename $LdCmd)"
+ SettingsMergeObjectsCommand="$(basename $MergeObjsCmd)"
+ SettingsMergeObjectsFlags="$MergeObjsArgs"
SettingsArCommand="$(basename $ArCmd)"
SettingsDllWrapCommand="$(basename $DllWrapCmd)"
SettingsWindresCommand="$(basename $WindresCmd)"
@@ -538,6 +544,8 @@ AC_DEFUN([FP_SETTINGS],
SettingsHaskellCPPCommand="$HaskellCPPCmd"
SettingsHaskellCPPFlags="$HaskellCPPArgs"
SettingsLdCommand="$LdCmd"
+ SettingsMergeObjectsCommand="$MergeObjsCmd"
+ SettingsMergeObjectsFlags="$MergeObjsArgs"
SettingsArCommand="$ArCmd"
SettingsRanlibCommand="$RanlibCmd"
if test -z "$DllWrapCmd"
@@ -578,6 +586,18 @@ AC_DEFUN([FP_SETTINGS],
else
SettingsOptCommand="$OptCmd"
fi
+ if test -z "$OtoolCmd"
+ then
+ SettingsOtoolCommand="otool"
+ else
+ SettingsOtoolCommand="$OtoolCmd"
+ fi
+ if test -z "$InstallNameToolCmd"
+ then
+ SettingsInstallNameToolCommand="install_name_tool"
+ else
+ SettingsInstallNameToolCommand="$InstallNameToolCmd"
+ fi
SettingsCCompilerFlags="$CONF_CC_OPTS_STAGE2"
SettingsCxxCompilerFlags="$CONF_CXX_OPTS_STAGE2"
SettingsCCompilerLinkFlags="$CONF_GCC_LINKER_OPTS_STAGE2"
@@ -592,8 +612,12 @@ AC_DEFUN([FP_SETTINGS],
AC_SUBST(SettingsCCompilerSupportsNoPie)
AC_SUBST(SettingsLdCommand)
AC_SUBST(SettingsLdFlags)
+ AC_SUBST(SettingsMergeObjectsCommand)
+ AC_SUBST(SettingsMergeObjectsFlags)
AC_SUBST(SettingsArCommand)
AC_SUBST(SettingsRanlibCommand)
+ AC_SUBST(SettingsOtoolCommand)
+ AC_SUBST(SettingsInstallNameToolCommand)
AC_SUBST(SettingsDllWrapCommand)
AC_SUBST(SettingsWindresCommand)
AC_SUBST(SettingsLibtoolCommand)
@@ -2589,7 +2613,7 @@ AC_DEFUN([CHECK_FOR_GOLD_T22266],[
])
$CC -c -o conftest.a.o conftest.a.c || AC_MSG_ERROR([Failed to compile test])
- $SettingsMergeObjectsCommand $SettingsMergeObjectsFlags -T conftest.t conftest.a.o -o conftest.ar.o || AC_MSG_ERROR([Failed to merge test object])
+ $MergeObjsCmd $MergeObjsArgs -T conftest.t conftest.a.o -o conftest.ar.o || AC_MSG_ERROR([Failed to merge test object])
$CC -c -o conftest.main.o conftest.main.c || AC_MSG_ERROR([Failed to compile test driver])
$CC conftest.ar.o conftest.main.o -o conftest || AC_MSG_ERROR([Failed to link test driver])
@@ -2609,33 +2633,30 @@ AC_DEFUN([CHECK_FOR_GOLD_T22266],[
# ------------------
# Find which linker to use to merge object files.
#
+# See Note [Merging object files for GHCi] in GHC.Driver.Pipeline.
AC_DEFUN([FIND_MERGE_OBJECTS],[
AC_REQUIRE([FIND_LD])
- if test -z "$SettingsMergeObjectsCommand"; then
- SettingsMergeObjectsCommand="$LD"
+ if test -z "$MergeObjsCmd"; then
+ MergeObjsCmd="$LD"
fi
- if test -z "$SettingsMergeObjectsFlags"; then
- SettingsMergeObjectsFlags="-r"
+ if test -z "$MergeObjsArgs"; then
+ MergeObjsArgs="-r"
fi
- CHECK_FOR_GOLD_T22266($SettingsMergeObjectsCommand)
+ CHECK_FOR_GOLD_T22266($MergeObjsCmd)
if test "$result" = "1"; then
- AC_MSG_NOTICE([$SettingsMergeObjectsCommand is broken due to binutils 22266, looking for another linker...])
- SettingsMergeObjectsCommand=""
- AC_CHECK_TARGET_TOOL([SettingsMergeObjectsCommand], [ld])
- CHECK_FOR_GOLD_T22266($SettingsMergeObjectsCommand)
+ AC_MSG_NOTICE([$MergeObjsCmd is broken due to binutils 22266, looking for another linker...])
+ MergeObjsCmd=""
+ AC_CHECK_TARGET_TOOL([MergeObjsCmd], [ld])
+ CHECK_FOR_GOLD_T22266($MergeObjsCmd)
if test "$result" = "1"; then
- AC_MSG_ERROR([Linker is affected by binutils 22266 but couldn't find another unaffected linker. Please set the SettingsMergeObjectsCommand variable to a functional linker.])
+ AC_MSG_ERROR([Linker is affected by binutils 22266 but couldn't find another unaffected linker. Please set the MergeObjsCmd variable to a functional linker.])
fi
fi
- if test "$windows" = YES -a "$EnableDistroToolchain" = "NO" -a "$WORD_SIZE" = 64; then
- SettingsMergeObjectsFlags="$SettingsMergeObjectsFlags --oformat=pe-bigobj-x86-64"
- fi
-
- AC_SUBST(SettingsMergeObjectsCommand)
- AC_SUBST(SettingsMergeObjectsFlags)
+ AC_SUBST([MergeObjsCmd])
+ AC_SUBST([MergeObjsArgs])
])
# FIND_PYTHON
=====================================
compiler/ghci/Linker.hs
=====================================
@@ -913,20 +913,22 @@ dynLoadObjs hsc_env pls at PersistentLinkerState{..} objs = do
ldInputs =
concatMap (\l -> [ Option ("-l" ++ l) ])
(nub $ snd <$> temp_sos)
- ++ concatMap (\lp -> [ Option ("-L" ++ lp)
- , Option "-Xlinker"
- , Option "-rpath"
- , Option "-Xlinker"
- , Option lp ])
+ ++ concatMap (\lp -> Option ("-L" ++ lp)
+ : if gopt Opt_RPath dflags
+ then [ Option "-Xlinker"
+ , Option "-rpath"
+ , Option "-Xlinker"
+ , Option lp ]
+ else [])
(nub $ fst <$> temp_sos)
++ concatMap
- (\lp ->
- [ Option ("-L" ++ lp)
- , Option "-Xlinker"
- , Option "-rpath"
- , Option "-Xlinker"
- , Option lp
- ])
+ (\lp -> Option ("-L" ++ lp)
+ : if gopt Opt_RPath dflags
+ then [ Option "-Xlinker"
+ , Option "-rpath"
+ , Option "-Xlinker"
+ , Option lp ]
+ else [])
minus_big_ls
-- See Note [-Xlinker -rpath vs -Wl,-rpath]
++ map (\l -> Option ("-l" ++ l)) minus_ls,
=====================================
compiler/main/DriverPipeline.hs
=====================================
@@ -382,7 +382,56 @@ compileEmptyStub dflags hsc_env basename location mod_name = do
-- ---------------------------------------------------------------------------
-- Link
-
+--
+-- Note [Dynamic linking on macOS]
+-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+--
+-- Since macOS Sierra (10.14), the dynamic system linker enforces
+-- a limit on the Load Commands. Specifically the Load Command Size
+-- Limit is at 32K (32768). The Load Commands contain the install
+-- name, dependencies, runpaths, and a few other commands. We however
+-- only have control over the install name, dependencies and runpaths.
+--
+-- The install name is the name by which this library will be
+-- referenced. This is such that we do not need to bake in the full
+-- absolute location of the library, and can move the library around.
+--
+-- The dependency commands contain the install names from of referenced
+-- libraries. Thus if a libraries install name is @rpath/libHS...dylib,
+-- that will end up as the dependency.
+--
+-- Finally we have the runpaths, which informs the linker about the
+-- directories to search for the referenced dependencies.
+--
+-- The system linker can do recursive linking, however using only the
+-- direct dependencies conflicts with ghc's ability to inline across
+-- packages, and as such would end up with unresolved symbols.
+--
+-- Thus we will pass the full dependency closure to the linker, and then
+-- ask the linker to remove any unused dynamic libraries (-dead_strip_dylibs).
+--
+-- We still need to add the relevant runpaths, for the dynamic linker to
+-- lookup the referenced libraries though. The linker (ld64) does not
+-- have any option to dead strip runpaths; which makes sense as runpaths
+-- can be used for dependencies of dependencies as well.
+--
+-- The solution we then take in GHC is to not pass any runpaths to the
+-- linker at link time, but inject them after the linking. For this to
+-- work we'll need to ask the linker to create enough space in the header
+-- to add more runpaths after the linking (-headerpad 8000).
+--
+-- After the library has been linked by $LD (usually ld64), we will use
+-- otool to inspect the libraries left over after dead stripping, compute
+-- the relevant runpaths, and inject them into the linked product using
+-- the install_name_tool command.
+--
+-- This strategy should produce the smallest possible set of load commands
+-- while still retaining some form of relocatability via runpaths.
+--
+-- The only way I can see to reduce the load command size further would be
+-- by shortening the library names, or start putting libraries into the same
+-- folders, such that one runpath would be sufficient for multiple/all
+-- libraries.
link :: GhcLink -- interactive or batch
-> DynFlags -- dynamic flags
-> Bool -- attempt linking in batch mode?
@@ -1769,9 +1818,12 @@ linkBinary' staticLink dflags o_files dep_packages = do
rc_objs <- maybeCreateManifest dflags output_fn
- let link = if staticLink
- then SysTools.runLibtool
- else SysTools.runLink
+ let link dflags args | staticLink = SysTools.runLibtool dflags args
+ | platformOS platform == OSDarwin
+ = SysTools.runLink dflags args >> SysTools.runInjectRPaths dflags pkg_lib_paths output_fn
+ | otherwise
+ = SysTools.runLink dflags args
+
link dflags (
map SysTools.Option verbFlags
++ [ SysTools.Option "-o"
@@ -1838,7 +1890,13 @@ linkBinary' staticLink dflags o_files dep_packages = do
++ pkg_link_opts
++ pkg_framework_opts
++ (if platformOS platform == OSDarwin
- then [ "-Wl,-dead_strip_dylibs" ]
+ -- dead_strip_dylibs, will remove unused dylibs, and thus save
+ -- space in the load commands. The -headerpad is necessary so
+ -- that we can inject more @rpath's later for the left over
+ -- libraries during runInjectRpaths phase.
+ --
+ -- See Note [Dynamic linking on macOS].
+ then [ "-Wl,-dead_strip_dylibs", "-Wl,-headerpad,8000" ]
else [])
))
=====================================
compiler/main/DynFlags.hs
=====================================
@@ -155,8 +155,8 @@ module DynFlags (
versionedAppDir, versionedFilePath,
extraGccViaCFlags, systemPackageConfig,
pgm_L, pgm_P, pgm_F, pgm_c, pgm_a, pgm_l, pgm_lm, pgm_dll, pgm_T,
- pgm_windres, pgm_libtool, pgm_ar, pgm_ranlib, pgm_lo, pgm_lc,
- pgm_lcc, pgm_i,
+ pgm_windres, pgm_libtool, pgm_ar, pgm_otool, pgm_install_name_tool,
+ pgm_ranlib, pgm_lo, pgm_lc, pgm_lcc, pgm_i,
opt_L, opt_P, opt_F, opt_c, opt_cxx, opt_a, opt_l, opt_lm, opt_i,
opt_P_signature,
opt_windres, opt_lo, opt_lc, opt_lcc,
@@ -1463,6 +1463,10 @@ pgm_lcc :: DynFlags -> (String,[Option])
pgm_lcc dflags = toolSettings_pgm_lcc $ toolSettings dflags
pgm_ar :: DynFlags -> String
pgm_ar dflags = toolSettings_pgm_ar $ toolSettings dflags
+pgm_otool :: DynFlags -> String
+pgm_otool dflags = toolSettings_pgm_otool $ toolSettings dflags
+pgm_install_name_tool :: DynFlags -> String
+pgm_install_name_tool dflags = toolSettings_pgm_install_name_tool $ toolSettings dflags
pgm_ranlib :: DynFlags -> String
pgm_ranlib dflags = toolSettings_pgm_ranlib $ toolSettings dflags
pgm_lo :: DynFlags -> (String,[Option])
@@ -3089,6 +3093,10 @@ dynamic_flags_deps = [
$ hasArg $ \f -> alterToolSettings $ \s -> s { toolSettings_pgm_libtool = f }
, make_ord_flag defFlag "pgmar"
$ hasArg $ \f -> alterToolSettings $ \s -> s { toolSettings_pgm_ar = f }
+ , make_ord_flag defFlag "pgmotool"
+ $ hasArg $ \f -> alterToolSettings $ \s -> s { toolSettings_pgm_otool = f}
+ , make_ord_flag defFlag "pgminstall_name_tool"
+ $ hasArg $ \f -> alterToolSettings $ \s -> s { toolSettings_pgm_install_name_tool = f}
, make_ord_flag defFlag "pgmranlib"
$ hasArg $ \f -> alterToolSettings $ \s -> s { toolSettings_pgm_ranlib = f }
@@ -4589,7 +4597,6 @@ defaultFlags settings
Opt_OmitYields,
Opt_PrintBindContents,
Opt_ProfCountEntries,
- Opt_RPath,
Opt_SharedImplib,
Opt_SimplPreInlining,
Opt_VersionMacros
@@ -4600,6 +4607,8 @@ defaultFlags settings
++ default_PIC platform
+ ++ default_RPath platform
+
++ concatMap (wayGeneralFlags platform) (defaultWays settings)
++ validHoleFitDefaults
@@ -4640,6 +4649,29 @@ default_PIC platform =
-- information.
_ -> []
+
+-- We usually want to use RPath, except on macOS (OSDarwin). On recent macOS
+-- versions the number of load commands we can embed in a dynamic library is
+-- restricted. Hence since b592bd98ff2 we rely on -dead_strip_dylib to only
+-- link the needed dylibs instead of linking the full dependency closure.
+--
+-- If we split the library linking into injecting -rpath and -l @rpath/...
+-- components, we will reduce the number of libraries we link, however we will
+-- still inject one -rpath entry for each library, independent of their use.
+-- That is, we even inject -rpath values for libraries that we dead_strip in
+-- the end. As such we can run afoul of the load command size limit simply
+-- by polluting the load commands with RPATH entries.
+--
+-- Thus, we disable Opt_RPath by default on OSDarwin. The savvy user can always
+-- enable it with -use-rpath if they so wish.
+--
+-- See Note [Dynamic linking on macOS]
+
+default_RPath :: Platform -> [GeneralFlag]
+default_RPath platform | platformOS platform == OSDarwin = []
+default_RPath _ = [Opt_RPath]
+
+
-- General flags that are switched on/off when other general flags are switched
-- on
impliedGFlags :: [(GeneralFlag, TurnOnFlag, GeneralFlag)]
=====================================
compiler/main/Settings.hs
=====================================
@@ -25,6 +25,8 @@ module Settings
, sPgm_windres
, sPgm_libtool
, sPgm_ar
+ , sPgm_otool
+ , sPgm_install_name_tool
, sPgm_ranlib
, sPgm_lo
, sPgm_lc
@@ -139,6 +141,10 @@ sPgm_libtool :: Settings -> String
sPgm_libtool = toolSettings_pgm_libtool . sToolSettings
sPgm_ar :: Settings -> String
sPgm_ar = toolSettings_pgm_ar . sToolSettings
+sPgm_otool :: Settings -> String
+sPgm_otool = toolSettings_pgm_otool . sToolSettings
+sPgm_install_name_tool :: Settings -> String
+sPgm_install_name_tool = toolSettings_pgm_install_name_tool . sToolSettings
sPgm_ranlib :: Settings -> String
sPgm_ranlib = toolSettings_pgm_ranlib . sToolSettings
sPgm_lo :: Settings -> (String, [Option])
=====================================
compiler/main/SysTools.hs
=====================================
@@ -254,7 +254,10 @@ linkDynLib dflags0 o_files dep_packages
| ( osElfTarget (platformOS (targetPlatform dflags)) ||
osMachOTarget (platformOS (targetPlatform dflags)) ) &&
dynLibLoader dflags == SystemDependent &&
- WayDyn `elem` ways dflags
+ -- Only if we want dynamic libraries
+ WayDyn `elem` ways dflags &&
+ -- Only use RPath if we explicitly asked for it
+ gopt Opt_RPath dflags
= ["-L" ++ l, "-Xlinker", "-rpath", "-Xlinker", l]
-- See Note [-Xlinker -rpath vs -Wl,-rpath]
| otherwise = ["-L" ++ l]
@@ -377,8 +380,15 @@ linkDynLib dflags0 o_files dep_packages
++ map Option pkg_lib_path_opts
++ map Option pkg_link_opts
++ map Option pkg_framework_opts
- ++ [ Option "-Wl,-dead_strip_dylibs" ]
+ -- dead_strip_dylibs, will remove unused dylibs, and thus save
+ -- space in the load commands. The -headerpad is necessary so
+ -- that we can inject more @rpath's later for the leftover
+ -- libraries in the runInjectRpaths phase below.
+ --
+ -- See Note [Dynamic linking on macOS]
+ ++ [ Option "-Wl,-dead_strip_dylibs", Option "-Wl,-headerpad,8000" ]
)
+ runInjectRPaths dflags pkg_lib_paths output_fn
_ -> do
-------------------------------------------------------------------
-- Making a DSO
=====================================
compiler/main/SysTools/Settings.hs
=====================================
@@ -119,6 +119,8 @@ initSettings top_dir = do
windres_path <- getToolSetting "windres command"
libtool_path <- getToolSetting "libtool command"
ar_path <- getToolSetting "ar command"
+ otool_path <- getToolSetting "otool command"
+ install_name_tool_path <- getToolSetting "install_name_tool command"
ranlib_path <- getToolSetting "ranlib command"
-- TODO this side-effect doesn't belong here. Reading and parsing the settings
@@ -141,7 +143,7 @@ initSettings top_dir = do
as_args = map Option cc_args
ld_prog = cc_prog
ld_args = map Option (cc_args ++ words cc_link_args_str)
- ld_r_prog <- getSetting "Merge objects command"
+ ld_r_prog <- getToolSetting "Merge objects command"
ld_r_args <- getSetting "Merge objects flags"
llvmTarget <- getSetting "LLVM target"
@@ -210,6 +212,8 @@ initSettings top_dir = do
, toolSettings_pgm_windres = windres_path
, toolSettings_pgm_libtool = libtool_path
, toolSettings_pgm_ar = ar_path
+ , toolSettings_pgm_otool = otool_path
+ , toolSettings_pgm_install_name_tool = install_name_tool_path
, toolSettings_pgm_ranlib = ranlib_path
, toolSettings_pgm_lo = (lo_prog,[])
, toolSettings_pgm_lc = (lc_prog,[])
=====================================
compiler/main/SysTools/Tasks.hs
=====================================
@@ -28,6 +28,10 @@ import LlvmCodeGen.Base (LlvmVersion, llvmVersionStr, supportedLlvmVersion, pars
import SysTools.Process
import SysTools.Info
+import Control.Monad (join, forM, filterM)
+import System.Directory (doesFileExist)
+import System.FilePath ((</>))
+
{-
************************************************************************
* *
@@ -237,6 +241,41 @@ figureLlvmVersion dflags = traceToolCommand dflags "llc" $ do
return Nothing)
+-- | On macOS we rely on the linkers @-dead_strip_dylibs@ flag to remove unused
+-- libraries from the dynamic library. We do this to reduce the number of load
+-- commands that end up in the dylib, and has been limited to 32K (32768) since
+-- macOS Sierra (10.14).
+--
+-- @-dead_strip_dylibs@ does not dead strip @-rpath@ entries, as such passing
+-- @-l@ and @-rpath@ to the linker will result in the unnecesasry libraries not
+-- being included in the load commands, however the @-rpath@ entries are all
+-- forced to be included. This can lead to 100s of @-rpath@ entries being
+-- included when only a handful of libraries end up being truely linked.
+--
+-- Thus after building the library, we run a fixup phase where we inject the
+-- @-rpath@ for each found library (in the given library search paths) into the
+-- dynamic library through @-add_rpath at .
+--
+-- See Note [Dynamic linking on macOS]
+runInjectRPaths :: DynFlags -> [FilePath] -> FilePath -> IO ()
+runInjectRPaths dflags lib_paths dylib = do
+ info <- lines <$> askOtool dflags Nothing [Option "-L", Option dylib]
+ -- filter the output for only the libraries. And then drop the @rpath prefix.
+ let libs = fmap (drop 7) $ filter (isPrefixOf "@rpath") $ fmap (head.words) $ info
+ -- find any pre-existing LC_PATH items
+ info <- fmap words.lines <$> askOtool dflags Nothing [Option "-l", Option dylib]
+ let paths = concatMap f info
+ where f ("path":p:_) = [p]
+ f _ = []
+ lib_paths' = [ p | p <- lib_paths, not (p `elem` paths) ]
+ -- only find those rpaths, that aren't already in the library.
+ rpaths <- nub.sort.join <$> forM libs (\f -> filterM (\l -> doesFileExist (l </> f)) lib_paths')
+ -- inject the rpaths
+ case rpaths of
+ [] -> return ()
+ _ -> runInstallNameTool dflags $ map Option $ "-add_rpath":(intersperse "-add_rpath" rpaths) ++ [dylib]
+
+
runLink :: DynFlags -> [Option] -> IO ()
runLink dflags args = traceToolCommand dflags "linker" $ do
-- See Note [Run-time linker info]
@@ -335,6 +374,17 @@ askAr dflags mb_cwd args = traceToolCommand dflags "ar" $ do
runSomethingWith dflags "Ar" ar args $ \real_args ->
readCreateProcessWithExitCode' (proc ar real_args){ cwd = mb_cwd }
+askOtool :: DynFlags -> Maybe FilePath -> [Option] -> IO String
+askOtool dflags mb_cwd args = do
+ let otool = pgm_otool dflags
+ runSomethingWith dflags "otool" otool args $ \real_args ->
+ readCreateProcessWithExitCode' (proc otool real_args){ cwd = mb_cwd }
+
+runInstallNameTool :: DynFlags -> [Option] -> IO ()
+runInstallNameTool dflags args = do
+ let tool = pgm_install_name_tool dflags
+ runSomethingFiltered dflags id "Install Name Tool" tool args Nothing Nothing
+
runRanlib :: DynFlags -> [Option] -> IO ()
runRanlib dflags args = traceToolCommand dflags "ranlib" $ do
let ranlib = pgm_ranlib dflags
=====================================
compiler/main/ToolSettings.hs
=====================================
@@ -31,6 +31,8 @@ data ToolSettings = ToolSettings
, toolSettings_pgm_windres :: String
, toolSettings_pgm_libtool :: String
, toolSettings_pgm_ar :: String
+ , toolSettings_pgm_otool :: String
+ , toolSettings_pgm_install_name_tool :: String
, toolSettings_pgm_ranlib :: String
, -- | LLVM: opt llvm optimiser
toolSettings_pgm_lo :: (String, [Option])
=====================================
configure.ac
=====================================
@@ -429,6 +429,8 @@ then
NM="${mingwbin}nm.exe"
RANLIB="${mingwbin}ranlib.exe"
OBJDUMP="${mingwbin}objdump.exe"
+ MergeObjsCmd="$LD"
+ MergeObjsArgs="-r --oformat=pe-bigobj-x86-64"
fp_prog_ar="${mingwbin}ar.exe"
AC_PATH_PROG([Genlib],[genlib])
@@ -669,6 +671,18 @@ else
fi
AC_SUBST([LibtoolCmd])
+dnl ** Which otool to use on macOS
+dnl --------------------------------------------------------------
+AC_CHECK_TARGET_TOOL([OTOOL], [otool])
+OtoolCmd="$OTOOL"
+AC_SUBST(OtoolCmd)
+
+dnl ** Which install_name_tool to use on macOS
+dnl --------------------------------------------------------------
+AC_CHECK_TARGET_TOOL([INSTALL_NAME_TOOL], [install_name_tool])
+InstallNameToolCmd="$INSTALL_NAME_TOOL"
+AC_SUBST(InstallNameToolCmd)
+
# Here is where we re-target which specific version of the LLVM
# tools we are looking for. In the past, GHC supported a number of
# versions of LLVM simultaneously, but that stopped working around
@@ -1479,6 +1493,8 @@ echo "\
libtool : $LibtoolCmd
objdump : $ObjdumpCmd
ranlib : $RanlibCmd
+ otool : $OtoolCmd
+ install_name_tool : $InstallNameToolCmd
windres : $WindresCmd
dllwrap : $DllWrapCmd
genlib : $GenlibCmd
=====================================
docs/users_guide/phases.rst
=====================================
@@ -95,6 +95,24 @@ given compilation phase:
Use ⟨cmd⟩ as the pre-processor (with :ghc-flag:`-F` only).
+.. ghc-flag:: -pgmotool ⟨cmd⟩
+ :shortdesc: Use ⟨cmd⟩ as the program to inspect mach-o dylibs on macOS
+ :type: dynamic
+ :category: phase-programs
+
+ Use ⟨cmd⟩ as the program to inspect mach-o dynamic libraries and
+ executables to read the dynamic library dependencies. We will compute
+ the necessary ``runpath``s to embed for the dependencies based on the
+ result of the ``otool`` call.
+
+.. ghc-flag:: -pgminstall_name_tool ⟨cmd⟩
+ :shortdesc: Use ⟨cmd⟩ as the program to inject ``runpath`` into mach-o dylibs on macOS
+ :type: dynamic
+ :category: phase-programs
+
+ Use ⟨cmd⟩ as the program to inject ``runpath``s into mach-o dynamic
+ libraries and executables. As detected by the ``otool`` call.
+
.. ghc-flag:: -pgmwindres ⟨cmd⟩
:shortdesc: Use ⟨cmd⟩ as the program for embedding manifests on Windows.
:type: dynamic
=====================================
ghc.mk
=====================================
@@ -414,7 +414,7 @@ else # CLEANING
# programs such as GHC and ghc-pkg, that we do not assume the stage0
# compiler already has installed (or up-to-date enough).
-PACKAGES_STAGE0 = binary text transformers mtl parsec Cabal/Cabal hpc ghc-boot-th ghc-boot template-haskell ghc-heap ghci
+PACKAGES_STAGE0 = binary transformers mtl hpc ghc-boot-th ghc-boot template-haskell text parsec Cabal/Cabal ghc-heap ghci
ifeq "$(Windows_Host)" "NO"
PACKAGES_STAGE0 += terminfo
endif
@@ -441,14 +441,14 @@ PACKAGES_STAGE1 += process
PACKAGES_STAGE1 += hpc
PACKAGES_STAGE1 += pretty
PACKAGES_STAGE1 += binary
-PACKAGES_STAGE1 += text
PACKAGES_STAGE1 += transformers
PACKAGES_STAGE1 += mtl
-PACKAGES_STAGE1 += parsec
-PACKAGES_STAGE1 += Cabal/Cabal
PACKAGES_STAGE1 += ghc-boot-th
PACKAGES_STAGE1 += ghc-boot
PACKAGES_STAGE1 += template-haskell
+PACKAGES_STAGE1 += text
+PACKAGES_STAGE1 += parsec
+PACKAGES_STAGE1 += Cabal/Cabal
PACKAGES_STAGE1 += ghc-compact
PACKAGES_STAGE1 += ghc-heap
=====================================
hadrian/cfg/system.config.in
=====================================
@@ -14,6 +14,7 @@ hs-cpp = @HaskellCPPCmd@
ld = @LdCmd@
make = @MakeCmd@
nm = @NmCmd@
+merge-objects = @MergeObjsCmd@
objdump = @ObjdumpCmd@
ranlib = @REAL_RANLIB_CMD@
sphinx-build = @SPHINXBUILD@
@@ -114,10 +115,10 @@ conf-ld-linker-args-stage1 = @CONF_LD_LINKER_OPTS_STAGE1@
conf-ld-linker-args-stage2 = @CONF_LD_LINKER_OPTS_STAGE2@
conf-ld-linker-args-stage3 = @CONF_LD_LINKER_OPTS_STAGE3@
-conf-merge-objects-args-stage0 = @SettingsMergeObjectsFlags@
-conf-merge-objects-args-stage1 = @SettingsMergeObjectsFlags@
-conf-merge-objects-args-stage2 = @SettingsMergeObjectsFlags@
-conf-merge-objects-args-stage3 = @SettingsMergeObjectsFlags@
+conf-merge-objects-args-stage0 = @MergeObjsArgs@
+conf-merge-objects-args-stage1 = @MergeObjsArgs@
+conf-merge-objects-args-stage2 = @MergeObjsArgs@
+conf-merge-objects-args-stage3 = @MergeObjsArgs@
# Settings:
@@ -147,6 +148,8 @@ settings-merge-objects-command = @SettingsMergeObjectsCommand@
settings-merge-objects-flags = @SettingsMergeObjectsFlags@
settings-ar-command = @SettingsArCommand@
settings-ranlib-command = @SettingsRanlibCommand@
+settings-otool-command = @SettingsOtoolCommand@
+settings-install_name_tool-command = @SettingsInstallNameToolCommand@
settings-dll-wrap-command = @SettingsDllWrapCommand@
settings-windres-command = @SettingsWindresCommand@
settings-libtool-command = @SettingsLibtoolCommand@
=====================================
hadrian/src/Builder.hs
=====================================
@@ -313,7 +313,7 @@ systemBuilderPath builder = case builder of
Happy -> fromKey "happy"
HsCpp -> fromKey "hs-cpp"
Ld _ -> fromKey "ld"
- MergeObjects _ -> fromKey "settings-merge-objects-command"
+ MergeObjects _ -> fromKey "merge-objects"
Make _ -> fromKey "make"
Makeinfo -> fromKey "makeinfo"
Nm -> fromKey "nm"
=====================================
hadrian/src/Oracles/Setting.hs
=====================================
@@ -113,6 +113,8 @@ data SettingsFileSetting
| SettingsFileSetting_MergeObjectsFlags
| SettingsFileSetting_ArCommand
| SettingsFileSetting_RanlibCommand
+ | SettingsFileSetting_OtoolCommand
+ | SettingsFileSetting_InstallNameToolCommand
| SettingsFileSetting_DllWrapCommand
| SettingsFileSetting_WindresCommand
| SettingsFileSetting_LibtoolCommand
@@ -198,6 +200,8 @@ settingsFileSetting key = lookupValueOrError configFile $ case key of
SettingsFileSetting_MergeObjectsFlags -> "settings-merge-objects-flags"
SettingsFileSetting_ArCommand -> "settings-ar-command"
SettingsFileSetting_RanlibCommand -> "settings-ranlib-command"
+ SettingsFileSetting_OtoolCommand -> "settings-otool-command"
+ SettingsFileSetting_InstallNameToolCommand -> "settings-install_name_tool-command"
SettingsFileSetting_DllWrapCommand -> "settings-dll-wrap-command"
SettingsFileSetting_WindresCommand -> "settings-windres-command"
SettingsFileSetting_LibtoolCommand -> "settings-libtool-command"
=====================================
hadrian/src/Rules/Generate.hs
=====================================
@@ -294,6 +294,8 @@ generateSettings = do
, ("ar flags", expr $ lookupValueOrError configFile "ar-args")
, ("ar supports at file", expr $ yesNo <$> flag ArSupportsAtFile)
, ("ranlib command", expr $ settingsFileSetting SettingsFileSetting_RanlibCommand)
+ , ("otool command", expr $ settingsFileSetting SettingsFileSetting_OtoolCommand)
+ , ("install_name_tool command", expr $ settingsFileSetting SettingsFileSetting_InstallNameToolCommand)
, ("touch command", expr $ settingsFileSetting SettingsFileSetting_TouchCommand)
, ("dllwrap command", expr $ settingsFileSetting SettingsFileSetting_DllWrapCommand)
, ("windres command", expr $ settingsFileSetting SettingsFileSetting_WindresCommand)
=====================================
includes/ghc.mk
=====================================
@@ -228,6 +228,8 @@ $(includes_SETTINGS) : includes/Makefile | $$(dir $$@)/.
@echo ',("ar flags", "$(ArArgs)")' >> $@
@echo ',("ar supports at file", "$(ArSupportsAtFile)")' >> $@
@echo ',("ranlib command", "$(SettingsRanlibCommand)")' >> $@
+ @echo ',("otool command", "$(SettingsOtoolCommand)")' >> $@
+ @echo ',("install_name_tool command", "$(SettingsInstallNameToolCommand)")' >> $@
@echo ',("touch command", "$(SettingsTouchCommand)")' >> $@
@echo ',("dllwrap command", "$(SettingsDllWrapCommand)")' >> $@
@echo ',("windres command", "$(SettingsWindresCommand)")' >> $@
=====================================
libraries/text
=====================================
@@ -1 +1 @@
-Subproject commit c6768a2a07e94b8b26d0f0e53517773de1110ce2
+Subproject commit e07c14940c25f33fe5b282912d745d3a79dd4ade
=====================================
mk/config.mk.in
=====================================
@@ -515,6 +515,8 @@ SettingsLdFlags = @SettingsLdFlags@
SettingsMergeObjectsCommand = @SettingsMergeObjectsCommand@
SettingsMergeObjectsFlags = @SettingsMergeObjectsFlags@
SettingsArCommand = @SettingsArCommand@
+SettingsOtoolCommand = @SettingsOtoolCommand@
+SettingsInstallNameToolCommand = @SettingsInstallNameToolCommand@
SettingsRanlibCommand = @SettingsRanlibCommand@
SettingsDllWrapCommand = @SettingsDllWrapCommand@
SettingsWindresCommand = @SettingsWindresCommand@
@@ -556,6 +558,16 @@ LD_STAGE1 = $(LD)
LD_STAGE2 = $(LD)
LD_STAGE3 = $(LD)
+MERGE_OBJS_STAGE0 = @MergeObjsCmd@
+MERGE_OBJS_STAGE1 = @MergeObjsCmd@
+MERGE_OBJS_STAGE2 = @MergeObjsCmd@
+MERGE_OBJS_STAGE3 = @MergeObjsCmd@
+
+MERGE_OBJS_STAGE0_FLAGS = @MergeObjsArgs@
+MERGE_OBJS_STAGE1_FLAGS = @MergeObjsArgs@
+MERGE_OBJS_STAGE2_FLAGS = @MergeObjsArgs@
+MERGE_OBJS_STAGE3_FLAGS = @MergeObjsArgs@
+
# Cross-compiling options
# See Note [CrossCompiling vs Stage1Only]
CrossCompiling = @CrossCompiling@
=====================================
rts/Linker.c
=====================================
@@ -1371,23 +1371,6 @@ void freeObjectCode (ObjectCode *oc)
stgFree(oc);
}
-/* -----------------------------------------------------------------------------
-* Sets the initial status of a fresh ObjectCode
-*/
-static void setOcInitialStatus(ObjectCode* oc) {
- /* If a target has requested the ObjectCode not to be resolved then
- honor this requests. Usually this means the ObjectCode has not been
- initialized and can't be. */
- if (oc->status == OBJECT_DONT_RESOLVE)
- return;
-
- if (oc->archiveMemberName == NULL) {
- oc->status = OBJECT_NEEDED;
- } else {
- oc->status = OBJECT_LOADED;
- }
-}
-
ObjectCode*
mkOc( pathchar *path, char *image, int imageSize,
bool mapped, char *archiveMemberName, int misalignment ) {
@@ -1420,7 +1403,11 @@ mkOc( pathchar *path, char *image, int imageSize,
oc->archiveMemberName = NULL;
}
- setOcInitialStatus( oc );
+ if (oc->archiveMemberName == NULL) {
+ oc->status = OBJECT_NEEDED;
+ } else {
+ oc->status = OBJECT_LOADED;
+ }
oc->fileSize = imageSize;
oc->symbols = NULL;
@@ -1711,8 +1698,17 @@ HsInt loadOc (ObjectCode* oc)
# endif
#endif
- /* loaded, but not resolved yet, ensure the OC is in a consistent state */
- setOcInitialStatus( oc );
+ /* Loaded, but not resolved yet, ensure the OC is in a consistent state.
+ If a target has requested the ObjectCode not to be resolved then honor
+ this requests. Usually this means the ObjectCode has not been initialized
+ and can't be. */
+ if (oc->status != OBJECT_DONT_RESOLVE) {
+ if (oc->archiveMemberName == NULL) {
+ oc->status = OBJECT_NEEDED;
+ } else {
+ oc->status = OBJECT_LOADED;
+ }
+ }
IF_DEBUG(linker, debugBelch("loadOc: done.\n"));
return 1;
=====================================
rules/build-package-way.mk
=====================================
@@ -107,6 +107,7 @@ endif
endif
# Build the GHCi library
+# See Note [Merging object files for GHCi] in GHC.Driver.Pipeline.
ifneq "$(filter $3, v p)" ""
$1_$2_$3_GHCI_LIB = $1/$2/build/HS$$($1_$2_COMPONENT_ID).$$($3_osuf)
ifeq "$$($1_$2_BUILD_GHCI_LIB)" "YES"
@@ -116,7 +117,7 @@ BINDIST_LIBS += $$($1_$2_$3_GHCI_LIB)
endif
endif
$$($1_$2_$3_GHCI_LIB) : $$($1_$2_$3_HS_OBJS) $$($1_$2_$3_CMM_OBJS) $$($1_$2_$3_C_OBJS) $$($1_$2_$3_S_OBJS) $$($1_$2_EXTRA_OBJS) $$($1_$2_LD_SCRIPT)
- $$(call cmd,SettingsMergeObjectsCommand) $(SettingsMergeObjectsFlags) $$(if $$($1_$2_LD_SCRIPT),$$($1_$2_LD_SCRIPT_CMD) $$($1_$2_LD_SCRIPT)) -o $$@ $$(EXTRA_LD_LINKER_OPTS) $$($1_$2_$3_HS_OBJS) $$($1_$2_$3_CMM_OBJS) $$($1_$2_$3_C_OBJS) $$($1_$2_$3_S_OBJS) $$($1_$2_EXTRA_OBJS)
+ $$(call cmd,MERGE_OBJS_STAGE$4) $(MERGE_OBJS_STAGE$4_FLAGS) $$(if $$($1_$2_LD_SCRIPT),$$($1_$2_LD_SCRIPT_CMD) $$($1_$2_LD_SCRIPT)) -o $$@ $$(EXTRA_LD_LINKER_OPTS) $$($1_$2_$3_HS_OBJS) $$($1_$2_$3_CMM_OBJS) $$($1_$2_$3_C_OBJS) $$($1_$2_$3_S_OBJS) $$($1_$2_EXTRA_OBJS)
ifeq "$$($1_$2_BUILD_GHCI_LIB)" "YES"
# Don't bother making ghci libs for bootstrapping packages
ifneq "$4" "0"
=====================================
testsuite/tests/rts/all.T
=====================================
@@ -403,7 +403,7 @@ test('T16514', unless(opsys('mingw32'), skip), compile_and_run, ['T16514_c.cpp -
test('test-zeroongc', extra_run_opts('-DZ'), compile_and_run, ['-debug'])
test('T13676',
- [when(opsys('darwin') or opsys('mingw32'), expect_broken(17447)),
+ [when(opsys('mingw32'), expect_broken(17447)),
extra_files(['T13676.hs'])],
ghci_script, ['T13676.script'])
=====================================
utils/ghc-cabal/ghc.mk
=====================================
@@ -23,9 +23,9 @@ CABAL_CONSTRAINT := --constraint="Cabal == $(CABAL_DOTTED_VERSION)"
# macros is triggered by `-hide-all-packages`, so we have to explicitly
# enumerate all packages we need in scope.
ifeq "$(Windows_Host)" "YES"
-CABAL_BUILD_DEPS := ghc-prim base array transformers time containers bytestring deepseq process pretty directory filepath Win32
+CABAL_BUILD_DEPS := ghc-prim base array transformers time containers bytestring deepseq process pretty directory filepath Win32 template-haskell
else
-CABAL_BUILD_DEPS := ghc-prim base array transformers time containers bytestring deepseq process pretty directory filepath unix
+CABAL_BUILD_DEPS := ghc-prim base array transformers time containers bytestring deepseq process pretty directory filepath unix template-haskell
endif
ghc-cabal_DIST_BINARY_NAME = ghc-cabal$(exeext0)
@@ -75,7 +75,7 @@ $(ghc-cabal_DIST_BINARY): $(CABAL_LEXER_DEP) utils/ghc-cabal/Main.hs $(TOUCH_DEP
-ilibraries/filepath \
-ilibraries/hpc \
-ilibraries/mtl \
- -ilibraries/text \
+ -ilibraries/text/src \
libraries/text/cbits/cbits.c \
-Ilibraries/text/include \
-ilibraries/parsec/src \
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/e2c9625a82516f4fcd516218f782d7007357aa77...43f970499d053098d1928ce3921176192ec07d7a
--
View it on GitLab: https://gitlab.haskell.org/ghc/ghc/-/compare/e2c9625a82516f4fcd516218f782d7007357aa77...43f970499d053098d1928ce3921176192ec07d7a
You're receiving this email because of your account on gitlab.haskell.org.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-commits/attachments/20201014/d78d53ea/attachment-0001.html>
More information about the ghc-commits
mailing list