Static values language extension proposal

[Mathieu Boespflug]

Hi Austin,

this is very useful information, thanks. So it seems that the rts
linker is here to stay for a while longer still, at least because
there is no good alternative on Windows as of yet.

If I understand you correctly, on Windows dynamic linking is not an
option in part because of the number-of-exported-symbols limit, and
when linking statically one hits the same limit if stuff like
-optl-Wl,--export-dynamic is used. So at least on Windows, the only
way out is the rts linker. Supporting both linkers is certainly an
option. If I remember correctly, the issue Facundo found with the rts
linker is that to use it for looking up symbol addresses, you
apparently need to have an object file loaded twice, effectively (once
statically linked at build time, the second time through the rts
linker at runtime for doing the lookups). Maybe there's a way around
that, or that could be added? In any case for platforms with no
alternative, like Windows, I guess double loading is a tolerable price
to pay.

Best,

Mathieu

On Wed, Jan 29, 2014 at 12:11 PM, Austin Seipp <austin at well-typed.com> wrote:
> Hello,
>
> I'd just like to say I haven't gone over every discussion in this
> thread and had time to digest it all - I thought I would just
> highlight a minor technicality.
>
> On Fri, Jan 24, 2014 at 11:19 AM, Facundo Domínguez
> <facundo.dominguez at tweag.io> wrote:
>> Looking up static references
>> ====================
>>
>> `unstatic` is implemented as a function which finds a top-level value
>> from the `GlobalName`, otherwise it raises an exception. It crucially
>> relies on leveraging the system's dynamic linker, so out-of-the-box
>> only works with dynamically linked binaries (but see below).
>> `unstatic` proceeds as follows:
>>
>>   * Determines the name of the shared library from the package name
>> and the package version.
>>
>>   * Determines the symbol of the value by Z-Encoding the package name,
>> the module name and the value name.
>>
>>   * Uses the system's dynamic linker interface to obtain the address
>> of the symbol.
>>
>>   * Converts the symbol to a haskell value with `GHC.Prim.addrToAny#`
>>
>> In principle, only symbols in shared libraries can be found. However,
>> the dynamic linker is able to find symbols in modules that are linked
>> statically if GHC is fed with the option -optl-Wl,--export-dynamic. A
>> future enhancement could be to have GHC warn the user when modules
>> using the extension are linked statically and this option is not used
>> during linking.
>>
>> GHC only defines symbols for exported definitions in modules. So
>> unstatic won't be able to find the private bindings of a module. For
>> this sake, the implementation of static should in addition ensure that
>> the bindings it gets will appear in the symbol table when they are not
>> exported by their defining modules.
>
> Regarding -optl-Wl,--export-dynamic for static builds, and all that
> jazz - if I am understanding you right, note that Windows is a bit
> particular here, because there is a hard limit on the number of
> symbols allowed in a DLL. That means forcefully exporting *everything*
> could quickly get you to the symbol limit with the dynamic linker with
> large-ish applications (one exported function or data type may result
> in a handful of exported symbols created.) If you want to see the pain
> this has caused GHC itself, please see GHC bug #5987[1], which makes
> dynamic support on windows difficult - it's currently disabled now
> anyway.
>
> Furthermore, dynamic DLLs on Windows are a bit tricky anyway as the
> loader is fundamentally different from your typical ld.so (which there
> are ways around[2], but a bit nasty as you have to hack the COFF
> file.) Windows unfortunately isn't in an easy position here, but it's
> improving and it would be unfortunate to neglect it.
>
> This restriction does not exist with the static linker inside the RTS,
> so my suggestion, I guess, is that I'm inclined to want this to work
> for *both* static/dynamic configurations out of the box without
> hackery, if at all possible, which would be great for Windows users
> especially until the dynamic story is back up to scratch.
>
> [1] https://ghc.haskell.org/trac/ghc/ticket/5987
> [2] http://blog.omega-prime.co.uk/?p=138
>
>> As the application depends on shared libraries, now a tool to collect
>> these libraries would be required so they can be distributed together
>> with the executable binary when deploying a Cloud Haskell application
>> in a cluster. We won't delve further into this problem.
>
> And for any people interested in this - on Linux, a tool like
> patchelf[3] would help immensely for moving executables+their
> dependencies around in a 'bundle' style way.
>
> [3] http://nixos.org/patchelf.html
>
> --
> Regards,
>
> Austin Seipp, Haskell Consultant
> Well-Typed LLP, http://www.well-typed.com/



-- 
Mathieu Boespflug
Founder at http://tweag.io.