[GHC] #10322: In ghci object code loader, linking against the previous temp dylib is not enough on OS X

[GHC]

#10322: In ghci object code loader, linking against the previous temp dylib is not
enough on OS X
-------------------------------------+-------------------------------------
              Reporter:  rwbarton    |             Owner:
                  Type:  bug         |            Status:  new
              Priority:  high        |         Milestone:  7.10.2
             Component:  Compiler    |           Version:  7.10.1
              Keywords:              |  Operating System:  MacOS X
          Architecture:              |   Type of failure:  None/Unknown
  Unknown/Multiple                   |        Blocked By:
             Test Case:              |   Related Tickets:
              Blocking:              |
Differential Revisions:              |
-------------------------------------+-------------------------------------
 joelteon encountered this issue in a rather complicated program, and we
 worked out the cause over IRC.

 Suppose that ghci needs to sequentially load three modules A, B and C,
 where B refers to symbols from A and C refers to symbols from both A and
 B. (For example modules B, C and another module D all contain Template
 Haskell that refers to the previous module(s).) The object code linker
 currently works like this:

 - Link the module `A.o` into a dylib `ghc_1.dylib`
 - Link the module `B.o` against `ghc_1.dylib` into a new dylib
 `ghc_2.dylib`
 - Link the module `C.o` against `ghc_2.dylib` into a new dylib
 `ghc_3.dylib`

 As a result, `ghc_2.dylib` ends up with a NEEDED (or whatever it's called
 in Mach-O) entry for `ghc_1.dylib`, and `ghc_3.dylib` ends up with a
 NEEDED entry for `ghc_2.dylib`.

 However, this apparently does not satisfy the OS X `dlopen`
 implementation, which complains about a missing symbol `_A_foo` referenced
 by `ghc_3.dylib` and which is defined in `ghc_1.dylib`. Apparently the
 dynamic loader only checks the direct dependencies when trying to resolve
 undefined symbols.

 (The Linux dynamic loader seems to be perfectly happy to use an indirect
 dependency to resolve an undefined symbol. But I found out that the linker
 [https://sourceware.org/bugzilla/show_bug.cgi?id=10238 gold] has the same
 sort of behavior as the OS X dynamic loader. I don't know whether there is
 any standard here, but it seems that we cannot rely on the Linux dynamic
 loader's behavior.)

 Presumably the fix is to keep track of all the previous temporary dylibs
 (rather than just one in `last_temp_so`) and link against all of them when
 building a new temporary dylib. I'm slightly worried about quadratic
 behavior here, but I think it's unlikely to be an issue in practice.

 I have a reproducer at http://lpaste.net/808723564239781888 (which I'll
 add to the test suite next) and oherrala ran the test on an OS X system
 with the following results:
 {{{
 =====> Last(ghci) 1167 of 4480 [0, 0, 0]
 cd ./ghci/scripts && HC="/Users/oherrala/gits/ghc/inplace/bin/ghc-stage2"
 HC_OPTS="-dcore-lint -dcmm-lint -dno-debug-output -no-user-package-db
 -rtsopts -fno-warn-tabs -fno-ghci-history "
 "/Users/oherrala/gits/ghc/inplace/bin/ghc-stage2" -dcore-lint -dcmm-lint
 -dno-debug-output -no-user-package-db -rtsopts -fno-warn-tabs -fno-ghci-
 history  --interactive -v0 -ignore-dot-ghci +RTS -I0.1 -RTS
 <Last.script > Last.run.stdout 2> Last.run.stderr
 Actual stderr output differs from expected:
 --- /dev/null   2015-04-18 17:23:26.000000000 +0300
 +++ ./ghci/scripts/Last.run.stderr      2015-04-18 17:23:34.000000000
 +0300
 @@ -0,0 +1,9 @@
 +ghc-stage2: panic! (the 'impossible' happened)
 +  (GHC version 7.11.20150418 for x86_64-apple-darwin):
 +       Loading temp shared object failed:
 dlopen(/var/folders/64/90jfy8lj65bcm1k02syxz_l80000gn/T/ghc18812_0/libghc18812_12.dylib,
 5): Symbol not found: _LastA_a_closure
 +  Referenced from:
 /var/folders/64/90jfy8lj65bcm1k02syxz_l80000gn/T/ghc18812_0/libghc18812_12.dylib
 +  Expected in: flat namespace
 + in
 /var/folders/64/90jfy8lj65bcm1k02syxz_l80000gn/T/ghc18812_0/libghc18812_12.dylib
 +
 +Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
 +
 Actual stdout output differs from expected:
 --- ./ghci/scripts/Last.stdout  2015-04-18 16:26:55.000000000 +0300
 +++ ./ghci/scripts/Last.run.stdout      2015-04-18 17:23:34.000000000
 +0300
 @@ -1,3 +1,2 @@
  3
  4
 -7
 *** unexpected failure for Last(ghci)
 }}}

--
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/10322>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler