proptotype of make style dep stuff

[Duncan Coutts]

On Fri, 2007-10-26 at 10:20 -0700, Isaac Potoczny-Jones wrote:
> Duncan and others,
> 
> I have some high-level thoughts:
> 
> = Standalone Tool =
> 
> Something I've talked about for a long time (and a few times tried to
> get someone to implement) is a kind of EDSL for Setup files so that
> people can replace Make more easily by using the Setup files, and this
> seems like it could be a good start.  You could use Parsec as a model
> for building a little language for dependencies and actions for those
> dependencies.  Make is a mess because it's really hard to build any kind
> of abstractions using it; the main thing it does for you is the Make
> algorithm you've shown here, and everything else is really hard.
> Haskell would be much better at this :)

Aye, I've used make in Gtk2Hs and it's pretty good. The main thing that
lets it down is the bad language, well languages. The shell language
plus the limited functional language on top. That and make does not
handle dynamic dependencies well. So doing make properly would be a good
thing since the basic principle is sound.

A Cabal configure/build EDSL for the make actions and indeed for most of
Cabal would also be a good thing. Some people have given it some
consideration. Doing that well should help to reduce complexity and code
size in Cabal.

> Cabal is getting more and more complex, and I think it would be good to
> consider how you might separate this from Cabal.  Perhaps you could
> implement a make-a-like library that's pretty standalone, but would be
> designed for use by Cabal, and indeed, Cabal would probably use it
> internally, but it could also be used outside of the context of Cabal.

We do have to be careful about dependencies. And I think it might
actually be good to integrate this into Cabal directly and change some
of the existing imperative code in Cabal to work more in a dependency
driven style. I would hope the make code would be reusable if we design
it properly.

> = Pros & Cons of Cabal Complexity =
> 
> I've tended to resist adding complexity like RPMs to Cabal because I
> tend to think that smaller, standalone tools that can be elegantly
> composed is better.  The good part of composable standalone tools is
> that you can spread out the development effort among folks who are
> experts in their smaller tool.  You also can keep Cabal from being so
> complicated that it's hard to bootstrap packages, and so that the
> changes between versions aren't as painful.

Keeping the core closer to what is needed to bootstrap is a good point.
We can perhaps move some tasks to tools built on the Cabal library, like
cabal-install. For example there's no need for the sdist feature to be
in Cabal directly, we could move it into cabal-install, at least if we
make cabal-install the primare command line UI that developers use. That
way it can have more dependencies like, zlib and tar to be able to
create and unpack .tar.gz file.

I've also wondered if the command line UI could be completely separated,
though that could make bootstrapping too hard. It could certainly be
better separated internally in the Cabal library and that would make us
think more clearly about what interface Cabal the library provides,
rather than Cabal as a command line tool via Setup.hs.

> The downside of keeping Cabal simple is that you'd sorta like it to be
> the case that Cabal is all you need to build any package.  As packages
> and dependencies among packages get more complex (in part, because Cabal
> facilitates that, which is good) it's natural to make Cabal more complex
> as well to accommodate more and more needs of package authors.
> 
> My only point, really, is that you think about these trade-offs whenever
> adding new capabilities to Cabal.  Would it be better to have a
> standalone tool that Cabal depends on?

As I say, I'd go the other way and have a 'cabal' command line tool that
depends on the Cabal library and provides more convenience features to
developers and users. The cabal-install program is going in that
direction.

> = Goals of Dependency Chasing =
> 
> I've been a bit out of the loop on this, so I apologize if I'm
> re-covering old ground.
> 
> A simpler goal of a module-chasing system is just to replace the need
> for other-modules.  Other-modules is used in compilation (although ghc
> --make doesn't really need it) and for haddock and for making source
> tarballs, and probably for other things.

Yes. Note that although ghc --make does not need it, if you miss out any
module from other-modules you'll get nasty link errors later. Much
later, in a rather confusing way. So that'd be another thing we could
fix.

> Ideally, the user should be able to specify only "Exposed modules" for
> libraries and "main modules" for executables, and Cabal should derive
> the other-modules itself.  That's less pain for the user, and you can
> get around using ghc --make.  Well, using ghc --make isn't much of a
> problem, but other compilers without a --make flag could then be made to
> work.  Maybe this is implemented already?

Dependency chasing is essential to support pre-processors properly, like
c2hs, and also for compilers that do not implement their own dependency
chasing. It's also important for being able to do parallel builds which
will be increasingly important.

I also think it'll make it easier for us to support building a
collection of related packages in one go.

> At first this sounds really easy, but then I wonder, how can Cabal know
> whether a particular module in an import list is a part of a dependency,
> or actually a part of the package?  Can Cabal query ghc-pkg for that
> kind of information? 

Yes.

> Can Cabal assume that if it can find the module in the local directory
> tree that it is a part of the current package? 

Yes. That's ghc's behaviour that local modules shadow package modules.

> Can you think of another way to make this work?

We could do with making getting the information out of ghc-pkg more
efficient by allowing us to get all the information in one call. Simon
suggested something like ghc-pkg dump that just gives us all the info on
all packages in one go.

> = Overall =
> 
> Overall, this sounds really cool, and I encourage you all to keep at it!
>  I trust you to decide if adding this kind of complexity to Cabal is
> worthwhile, and to build a really nice system.  I really do think that
> Haskell should be able to do better than Make here, and it would be a
> pleasure to feel like we can replace bits of that toolchain :)

:-)

> I'll try to read in more detail and give more feedback.

Great.

Duncan