Dynamic Linking help

[Ben Gamari]

lonetiger at gmail.com writes:

> Hi Ben,
>
> Thanks for the reply!
>
Sure.

>> > Hi *,
>> >
>> > I’ve been working the past 4 or so months on reviving dynamic linking
>> > support for Windows in a way that has the most chance of working.
>> >
>> > My first patch in the series is up on Phabricator and with this patch
>> > dynamic linking work again, but only for the threaded RTS.
>> >
>> Thanks for all of your work on this, Tamar!
>> 
>
> Home stretch :)
>
Yay!

>> Hmm, why? I thought recent Windows releases had a notion of "user local"
>> installation, no? From what little I have heard it sounds like SxS
>> assemblies is the right solution here.
>
> Yes, so to be clear, SxS absolutely solve this problem. For final installs.
> The majority of the issue is that the testsuite won't have the assemblies in
> the SxS cache.
>
> There *is* a sort of RPATH equivalent for SxS that can be used here, it however
> has two problems:
> 1) Even though the API has no such limit, the implementation in the Windows loader
>    limits it per application to 5 entries.

 (╯°□°）╯︵ ┻━┻

>    Obviously this won't be enough. 
>    So this is absolutely another option (maybe even preferable now that I think about it
>    since it require almost no code change, mostly some build system changes.).
>    Can do either one of two things:
>    a) Copy all dll's to the lib folder when they're compiled instead of leaving them in place and
>       add a single SxS search entry to find them.
>    b) Turn of SxS in the testsuite for any Assemblies not the RTS, and add the inplace RTS directory to
>       the SxS search path. Since it's only the RTS that's an issue.
>       
> 2) The other problem is that the paths specified have to be relative to the application.
>    (Of the top of my head) It doesn't support absolute paths. Which
>    means I can't have GHC generate the entry because I have no idea
>    where the testsuite intends to run the binary.
>    One way around this is to have the testsuite generate the needed config file. That should be do-able.
>    
> I'll investigate this method first. I had discarded it for some reason before but now can't remember...
>
Right, it sounds like this is a workable option and the fact that it
requires adding no further complexity to the compiler is quite a merit.
The only question is what other use-cases might run into this same issue.
For instance, what happens when I run `cabal build` and try to run an
executable from `dist/build`. Then I run `cabal install` and run it from
`.cabal/bin`. Surely Cabal will need to take some sort of action in this
case. I suppose this means that using plain `ghc -dynamic` alone is
probably out of the question.

>> > This is a problem for running tests in the testsuite using the inplace GHC.
>> >
>> > Typically on Windows the way you would do this is by delay loading the
>> > dll. This allows me to write some code on startup and manually load
>> > the runtime dll. The Windows loader would then just use the loaded
>> > dll. Unfortunately delay loading does not support const extern data.
>> > Such as const extern RtsConfig defaultRtsConfig;
>> >
>> Silly Windows.
>
> Yeah, unfortunately this is because this isn't done by any OS
> functions. Lazy loading is purely something implemented by linkers on
> Windows and the appropriate runtimes. In essense it just creates a
> stub that replaces all functions, which first checks if the dll is
> loaded, if not load it and then call the real function. Which is why
> it only works for functions.
>
Ahhh. I see, that makes sense.
>> 
>> > The RTS/GHC is full of such usage so it won’t be easy to change.
>> > Though I’d only have to change those exposed by Rts.h.
>> >
>> > So I have two options:
>> > 1) Replace all const externs with a function call. This would work,
>> >    but isn’t ideal because it would break if someone in the future
>> >    adds a new data entry instead of a function. And we have an extra
>> >    function call everywhere.
>> 
>> Right, I'm really not a fan of this option. Crippling the RTS's use of C
>> on account of arbitrary limitations of the Windows dynamic linker
>> doesn't seem very appealing.
>
> I was not a fan of this either. Would imagine I would be chased around
> with flaming pitchforks were I to do this..
>
Yes, sharp, rusty, flaming pitchforks.

>> > 2) I could do some hacks on the Windows side, e.g. compile the program
>> > to a shared library, embed the shared library inside the exe and on
>> > startup after loading the propert rts, load the DLL from (mmapped)
>> > memory and run the code.
>> 
>> This sounds like it would get the job done, although it certainly adds
>> complexity. Do you have any intuition for how much work it would be to
>> implement this?
>
> We sorta already do 80% of this. For Windows when making a dynamic
> version of GHC, the ghc-stage2.exe is a very thin shell, who's only
> purpose is to load the right libraries and change the search path to
> include the lib folder. The actual code is in a dll named e.g.
> ghc-stage2.exe.dll. The change needed would be to embed this dll into
> the exe (which is trivially done), and then load it from memory. This
> would require some work but there are enough wrapper code out there
> with appropriate licenses that we can use to accomplish this. Or at
> least get a running head start.
>
Alright, good to know. We can keep this one in our back pocket in case
the option described about doesn't work out.

Cheers,

- Ben

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 454 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20161028/e3d2607e/attachment.sig>