Build system woes

[Roman Leschinskiy]

Hi all,

SimonPJ asked me to summarise the problems I've been having with GHC's  
new build system with the idea that we could perhaps talk a bit about  
this during today's IRC meeting. The context is the Data Parallel  
Haskell project and in particular the dph packages which need a number  
of not quite straightforward things from the build system (this is  
only going to get worse in the future).

A lot of the problems stem from Cabal's lack of extensibility. This  
means that whenever I need some new Cabal functionality for dph I have  
to ask Ian or Duncan to implement it. They both are very helpful but  
this just doesn't scale! IMO, at the moment Cabal is simply not mature  
enough for building complex projects.

But wait, we do have those Setup.hs files which are customisable so  
why don't I just do what I want with them? Firstly, they aren't  
customisable enough. And secondly, I have no idea how to do what I  
want. The development version of Cabal changes on a daily basis and  
isn't documented. For instance, the black magic that Ian implemented  
in dph/dph/Setup.hs a month ago (and which I don't really understand)  
has been broken once already. Fortunately, I was able to fix that  
myself but if something more complex changes, I'll have to ask Ian  
again. In effect, I'm not able to properly maintain my own library!

I don't think I understand how GHC itself is built any longer, either.  
What does cabal-bin do? What is runghc.wrapper? What is the difference  
between make.library.* and build.library.* and why do we need both  
(and why doesn't one of them work for dph)? It is important to have an  
idea how the build system works when hacking on a project but how do I  
find out?

And finally, there is the problem of the HEAD being broken quite  
frequently. I have decided to pull as little as possible and it turns  
out that others have independently arrived at the same conclusion.  
This is bad! For instance, I would have spotted the OS X breakage much  
earlier if I pulled more frequently; now, I have to wade through a  
couple of weeks' worth of patches to find the culprit (that darcs  
unpull doesn't work in my repo doesn't help).

IMO, the main issue is that we have two moving targets: the GHC build  
system and the tool it relies on (Cabal). Both change rapidly and  
there is just no chance, for me at least, to keep up. If I understand  
correctly, the situation is this:

  - Ultimately, Cabal will be used for building GHC.

  - At the moment, Cabal doesn't provide the necessary functionality.

The current approach seems to be to gradually implement stuff in Cabal  
and to immediately use it in the build system. I think this just  
doesn't work for a highly collaborative project such as GHC. I have  
two suggestions how to improve things.

  1. I think it would be better to implement, document and release all  
the necessary
     functionality in Cabal first and only then to modify GHC's build  
system to use this
     functionality. In effect, this means that the Cabal version used  
for building GHC
     would be frozen at the beginning of a development cycle and not  
at the end. Critical
     stuff could still be merged, perhaps, but that should happen very  
infrequently and
     be backwards compatible.

  2. Ian, it would be great if you could add a couple of lines to  
build system patches
     explaining what they do and why. This could ultimately mean less  
work for you
     because I will bug you less often :-)

Sorry for the long rant, I simply seem to spend too much time  
wrestling with the build system.

Roman