Problem with runProgram (Using a shared library for the C++ in wxhaskell)

[Dave Tapley]

On 8 December 2011 22:42, Dave Tapley <dukedave at gmail.com> wrote:
> Hi all, I've been working on the build system for wxhaskell, and ran
> in to a problem using "runProgram".
>
> As an experiment I've decided to put it on hpaste instead of having
> the mail thread get out of hand (I'm also cross posting it to the
> wxhaskell list and have mentioned it in #haskell, and I want to keep
> all correspondence in one place).
> Here is a link which jumps to the line where I first describe the
> problem I'm having:
>
> http://hpaste.org/55027#line59

Ha, well that was a good time for hpaste to go down!
Here it is:

I've been trying to resurrect the idea of building a shared library
for the C++ component of the wxhaskell library.

Jeremy (to my knowledge) first put this forward, here:
http://wewantarock.wordpress.com/2010/11/01/working-around-the-static-libstdc-restriction/

In addition to the advantages he lists there, I have the following reason:
When building wx-core (in its current incarnation) cabal will *always*
rebuild all the C++ code, which takes a majority of the build time.
This becomes very frustrating when you change one line of Haskell and
have to wait seven minutes to rebuild.
I complained about it here:
http://sourceforge.net/mailarchive/message.php?msg_id=28099997

So, I decided to try and stop this complete C++ rebuild, I partially
succeed, but I eventually got stuck for reasons I complained here:
http://www.haskell.org/pipermail/cabal-devel/2011-October/007816.html

Side note: I did eventually discover why (or, where) the c-sources
(cSources) list is used to compile and link, and it is here:
http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-GHC.html#buildLib
You can see:
"| filename <- cSources libBi]" under "-- build any C sources", and
you can see:
"cSharedObjs = map (`replaceExtension` ("dyn_" ++ objExtension))
(cSources libBi)"
under "-- link:".
Is this assumption correct?

At this point I thought about either:
1. Getting the cabal source and starting to write code to give
BuildInfo a "cObjs" in addition to "cSources".
2. Picking up Jeremy's shared library code, so wxhaskell would have
its own code to build the library, in which I could do sensible
re-compilation.

Given that there were other advantages to 2, I went with that.

Jeremy had written one blog post on building a such a shared library, here:
http://wewantarock.wordpress.com/2010/11/03/building-a-shared-library-in-cabal/
In response to an email on the wxhaskell-devel list he also kindly put
up this gist, with the code he'd been working on:
https://gist.github.com/1301115

I dutifully took this gist, and have now attempted to integrate in to
my wxhaskell-dev branch, which you can find here:
http://darcsden.com/DukeDave/wxhaskell-dev
(note: I haven't pushed any of the shared library code to darcsden
yet, for reasons I'm about to explain)

This is where things got interesting, after a few hours of hacking I
have built a shared library, but in this very back-handed way:

Firstly, in Jeremy's code the to-be-compiled shared library is added
to the wxcore BuildInfo through a custom hook.
We can see this because:
> let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"] custom_bi
(the modified wxcore.cabal contains the line: "x-dll-name: wxc")

However on a clean build of wxcore we get the following error:

setup: Missing dependency on a foreign library:
* Missing C library: wxc
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.

The error is dumped *before* cabal calls myBuildHook, and since this
actually builds the library the error makes sense; cabal is looking
for the shared library before we've built it.

To get around this I modified the offending line, thus:
> let all_dlls   = parseDLLs ["x-dll-extra-libraries"] custom_bi

With that modification cabal would now get to myBuildHook, but then
another curious error arose:
We see that the actual linking is called here:
> runProgram verbose gcc (opts' ++ objs' ++ lib_dirs' ++ libs')

But on hitting that line the following error is spat out:
/usr/bin/ld: cannot open output file  dist/build/libwxc.so.0.13.1: No
such file or directory

I checked all my permissions and couldn't see anything wrong, I could
touch the file.
Conveniently I noticed that, if the verbosity is set high enough,
runProgram will call printRawCommandAndArgs:
http://hackage.haskell.org/packages/archive/Cabal/latest/doc/html/src/Distribution-Simple-Utils.html#printRawCommandAndArgs

The output (i.e. the linker invocation) looks like this:
/usr/bin/gcc -fno-stack-protector -shared -Wl,-soname,libwxc.so.0 -o
dist/build/libwxc.so.0.13.1 dist/build/src/cpp/apppath.o
dist/build/src/cpp/dragimage.o dist/build/src/cpp/eljaccelerator.o
[snip-rest-of-.o-files] -L/usr/local/lib -lstdc++ -lwx_baseu-2.9
[snip-rest-of-wx-libs]

Now if I cd into the wxcore and paste the command *verbatim* then gcc
works and generates libwxc.so.0.13.1 as expected.
You can see in Jeremy's code the linkShareLib function contains:
> cwd <- getCurrentDirectory

I used this to confirm that the we were in ./wxcore and we are, even
making the path for -o absolute didn't sovle the issue.
I ended up replacing runProgram line with this, less satisfactory line:
> system $ (unwords ([show . locationPath . programLocation $ gcc] ++ opts' ++ objs' ++ lib_dirs' ++ libs'))

Which works, but still doesn't explain why runProgram doesn't.
Any suggestions?

So with that complete I was able to link a shared library to:
wxcore/dist/build/libwxc.so.0.13.1

Of course, at this point we've generated the shared library, but
wxcore still isn't aware of it, so the build completes successfully,
but any attempt to compile against it results (understandably) in
hundreds of linker errors.
To resolve this I had to add "x-dll-name" (read in from wxcore.cabal
as "wxc") back to parseDLLs:
> let all_dlls   = parseDLLs ["x-dll-name", "x-dll-extra-libraries"] custom_bi

Well, not quite, attempt to build wxcore again and we're back to this error:
* Missing C library: wxc

Of course, the .so isn't in a 'normal' place, so add its location as
an extra lib directory:
> { extraLibDirs = extraLibDirs libbi ++ extraLibDirs wx ++ ["/full/path/to/wxcore/dist/build"]
(note: I tried using just "dist/build", but cabal said: "library-dirs:
dist/build is a relative path")

This still isn't quite enough, it took one more kludge:
/full/path/to/wxcore/dist/build/$ ln -s
/full/path/to/wxcore/dist/build/libwxc.so.0.13.1 libwxc.so

I'd like to think there's a more elegant solution than this and I'm
open to suggestions.
I should note that at this point I became aware that we could have an
entirely separate cabal project (perhaps "wxc") for this shared
library, and then have wxcore depend on it?

Running one more time I *thought* I was finally there, until I noticed
this line, hidden in cabal's output:
/full/path/to/wxcore/dist/build/libwxc.so: file not recognized: File truncated

As far as I can tell, this occurs because we now load(?)
"libwxc.so.0.13.1" in myConfHook, but then in myBuildHook we want to
use it as the destination for the linker. Worse still it seems that
'truncated' actually means 'deleted'. To get around this I had to one
again remove "x-dll-name" from the parseDLLs call (so I could get to
myBuildHook and create "libwxc.so.0.13.1" again), build wxcore, then
add "x-dll-name" back again, but this time also comment out the call
to linkSharedLib (and so stop cabal attempting to open
"libwxc.so.0.13.1") before building wxcore one last time.

Finally everything seemed to work.

Now, I should note that I have been using cabal-dev, and so I now have
a package-conf I need to reference when building with my new shared
library wxhaskell. To test it I built the wxhaskell HelloWorld sample
thus:
/full/path/to/samples/wxcore$ ghc --make -package-conf
../../cabal-dev/packages-7.0.3.conf HelloWorld.hs

It worked, but then when I try to run HelloWorld I get:
./HelloWorld: error while loading shared libraries: libwxc.so.0:
cannot open shared object file: No such file or directory

Okay, again, libwxc.so.0 isn't in a 'normal' place, so I kludged it
with another symlink:
/full/path/to/samples/wxcore$ ln -s
/full/path/to/wxcore/dist/build/libwxc.so libwxc.so.0

And now, I can report that HelloWorld runs.
[imagine a screen shot of the HelloWorld sample application running here]
Rejoice.

But wait, there's more:
One of the promises of the shared library approach is that we'd all be
able to use wxhaskell with ghci again, and to my utter disbelief, it's
true:
/full/path/to/samples/wxcore$ ghci -package-conf
../../cabal-dev/packages-7.0.3.conf HelloWorld.hs
Ok, modules loaded: Main.
Prelude Main> main

[imagine a screen shot of the HelloWorld sample application running here]
Double rejoice.

>
> Thanks,
> Dave