LLVM and dynamic linking

[Aaron Friel]

I eagerly look forward to these patches, I hope they are able to land in time for the 7.8 release as well. Do you have any additional information on them - or is it part of a branch I could look at?

And I apologize for the polarizing tone - I’m overdramatizing the situation and I’m new to following GHC at the root (or head, whichever). Regardless, the LLVM dynamic linking issue has popped up now and again (there are a good number of trac issues) and I’m eager to see that GHC is able to be built properly with it and have it stay working.

I have no doubt the issues Ben and others have been working with are subtle and complex. There are absolutely brilliant people here working on GHC, so any problem left unsolved is bound to be uniquely difficult.

From: Carter Schonwald<mailto:carter.schonwald at gmail.com>
Sent: ‎Wednesday‎, ‎January‎ ‎1‎, ‎2014 ‎9‎:‎53‎ ‎PM
To: Aaron Friel<mailto:aaron at frieltek.com>
Cc: ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>

well, please feel welcome to ask for help as much as you need! To repeat: if you use ghc HEAD, it should be doable to build GHC head (using head as the bootstrap compiler) using LLVM. Once Ben's llvm dy linking patches land, you should be able to do both dynamic and static linking with  llvm.

As for your Mavericks example, if you review ghc trac and the mailing lists plus irc logs, it took the effort of several folks spread over several months to make sure that once Mavericks / Xcode 5 landed, that it would be "easy" to fix.

that said, theres no need to take such a polarizing tone, with speculations about the priorities of the various GHC devs. We're all volunteers  (ok, theres a some who are paid volunteers) who care about making sure ghc works as well as possible for everyone, but have finite time in the day, and so many different ways to ghc can be made better. (and in many cases, have a day job that also needs attention too).

please test things and holler when they don't work, and if you can debug problems and cook up good patches, great!

in the case of llvm and dynamic linking, the root cause was actually pretty darn subtle, and I'm immensely grateful that Ben Gamari got to the root of it. (I'd definitely hit the problem myself, and I was absolutely stumped when I tried to investigate it.)


On Wed, Jan 1, 2014 at 10:03 PM, Aaron Friel <aaron at frieltek.com<mailto:aaron at frieltek.com>> wrote:
Because I think it’s going to be an organizational issue and a duplication of effort if GHC is built one way but the future direction of LLVM is another.

Imagine if GCC started developing a new engine and it didn’t work with one of the biggest, most regular consumers of GCC. Say, the Linux kernel, or itself. At first, the situation is optimistic - if this engine doesn’t work for the project that has the smartest, brightest GCC hackers potentially looking at it, then it should fix itself soon enough. Suppose the situation lingers though, and continues for months without fix. The new GCC backend starts to become the default, and the community around GCC advocates for end-users to use it to optimize code for their projects and it even becomes the default for some platforms, such as ARM.

What I’ve described is analogous to the GHC situation - and the result is that GHC isn’t self-hosting on some platforms and the inertia that used to be behind the LLVM backend seems to have stagnated. Whereas LLVM used to be the “new hotness”, I’ve noticed that issues like Trac #7787 no longer have a lot of eyes on them and externally it seems like GHC has accepted a bifurcated approach for development.

I dramatize the situation above, but there’s some truth to it. The LLVM backend needs some care and attention and if the majority of GHC devs can’t build GHC with LLVM, then that means the smartest, brightest GHC hackers won’t have their attention turned toward fixing those problems. If a patch to GHC-HEAD broke compilation for every backend, it would be fixed in short order. If a new version of GCC did not work with GHC, I can imagine it would be only hours before the first patches came in resolving the issue. On OS X Mavericks, an incompatibility with GHC has led to a swift reaction and strong support for resolving platform issues. The attention to the LLVM backend is visibly smaller, but I don’t know enough about the people working on GHC to know if it is actually smaller.

The way I am trying to change this is by making it easier for people to start using GHC (by putting images on Docker.io) and, in the process, learning about GHC’s build process and trying to make things work for my own projects. The Docker image allows anyone with a Linux kernel to build and play with GHC HEAD. The information about building GHC yourself is difficult to approach and I found it hard to get started, and I want to improve that too, so I’m learning and asking questions.

From: Carter Schonwald<mailto:carter.schonwald at gmail.com>
Sent: ‎Wednesday‎, ‎January‎ ‎1‎, ‎2014 ‎5‎:‎54‎ ‎PM
To: Aaron Friel<mailto:aaron at frieltek.com>
Cc: ghc-devs at haskell.org<mailto:ghc-devs at haskell.org>

7.8 should have working dylib support on the llvm backend. (i believe some of the relevant patches are in head already, though Ben Gamari can opine on that)

why do you want ghc to be built with llvm? (i know i've tried myself in the past, and it should be doable with 7.8 using 7.8 soon too)


On Wed, Jan 1, 2014 at 5:38 PM, Aaron Friel <aaron at frieltek.com<mailto:aaron at frieltek.com>> wrote:
Replying to include the email list. You’re right, the llvm backend and the gmp licensing issues are orthogonal - or should be. The problem is I get build errors when trying to build GHC with LLVM and dynamic libraries.

The result is that I get a few different choices when producing a platform image for development, with some uncomfortable tradeoffs:


  1.
LLVM-built GHC, dynamic libs - doesn’t build.
  2.
LLVM-built GHC, static libs - potential licensing oddities with me shipping a statically linked ghc binary that is now gpled. I am not a lawyer, but the situation makes me uncomfortable.
  3.
GCC/ASM-built GHC, dynamic libs - this is the *standard* for most platforms shipping ghc binaries, but it means that one of the biggest and most critical users of the LLVM backend is neglecting it. It also bifurcates development resources for GHC. Optimization work is duplicated and already devs are getting into the uncomfortable position of suggesting to users that they should trust GHC to build your programs in a particular way, but not itself.
  4.
GCC/ASM-built GHC, static libs - worst of all possible worlds.

Because of this, the libgmp and llvm-backend issues aren’t entirely orthogonal. Trac ticket #7885 is exactly the issue I get when trying to compile #1.

From: Carter Schonwald<mailto:carter.schonwald at gmail.com>
Sent: ‎Monday‎, ‎December‎ ‎30‎, ‎2013 ‎1‎:‎05‎ ‎PM
To: Aaron Friel<mailto:aaron at frieltek.com>

Good question but you forgot to email the mailing list too :-)

Using llvm has nothing to do with Gmp. Use the native code gen (it's simper) and integer-simple.

That said, standard ghc dylinks to a system copy of Gmp anyways (I think ). Building ghc as a Dylib is orthogonal.

-Carter

On Dec 30, 2013, at 1:58 PM, Aaron Friel <aaron at frieltek.com<mailto:aaron at frieltek.com>> wrote:

Excellent research - I’m curious if this is the right thread to inquire about the status of trying to link GHC itself dynamically.

I’ve been attempting to do so with various LLVM versions (3.2, 3.3, 3.4) using snapshot builds of GHC (within the past week) from git, and I hit ticket #7885 [https://ghc.haskell.org/trac/ghc/ticket/7885] every time (even the exact same error message).

I’m interested in dynamically linking GHC with LLVM to avoid the entanglement with libgmp’s license.

If this is the wrong thread or if I should reply instead to the trac item, please let me know.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/ghc-devs/attachments/20140102/923c9d41/attachment-0001.html>