planning for ghc-6.10.1 and hackage [or: combining packages to yield new type correct programs]

[Don Stewart]

duncan.coutts:
> Hi,
> 
> We're getting pretty close to a final ghc-6.10.1 release. We would like
> of course for the transition this time to be less painful than last
> time. We all got a lot of flack last time for having no plan in place
> and making everyone change all their .cabal files etc.
> 
> This time we can do a lot better. We have already decided to ship two
> versions of base, version 3 and version 4. Version 4 is the even more
> stripped down one, and with changes to exception handling etc. The
> version 3.1 of base depends on base 4 and re-exports most of it with a
> few differences so that it provides essentially the same api as base 3.0
> did (which was included in ghc-6.8.x).
> 
> However we are not all the way there yet. If we released ghc-6.10.1
> today everything would still break. By everything I mean everyone's
> private projects and all the packages on hackage. The only packages that
> would build would be trivial ones, or the ones that come with ghc.

Here are my notes on why this is so hard, and so scary, and what we can do.

So Duncan and I spent about 6 hours tonight working out how to make the 
cabal-install constraint solver, and the Cabal configure invariants,
work when programs attempt to use base-3 and base-4 on the same system.

Here's a summary of why this is non-trivial,

* We're trying to compose packages on the users machine to yield new
  type correct programs.

* We're using cabal dependencies to decide when it is safe to do this.
  Hopefully we don't rule in any type incorrect combinations, nor rule
  out to many type correct combinations.

* This is scary - in fact, we think the package system admits type
  incorrect programs (based on Typeable, or initialised global state), 
  as it is similar to the runtime linking problem for modules.

* We use constraint solving determine when composition is safe, by looking at 
        "package >= 3 && < 4"
  style constraints. That is, we try to guess when the composition
  would yield a type correct program.

* Again, we're using constraint solving on this language to
  determine when composition of Haskell module sets (aka packages) would
  yield type correct Haskell programs
  
  All without attempting to do type checking of the interfaces between
  packages -- the very thing that says whether this is sound!

* This previously relied on a strong invariant:

    + There was only to be one package-version assignment in a program
    + People didn't change APIs too much, and tried to follow the
      versioning spec, and wrote .cabal deps that followed the spec.

* But now we allow multiple package-version assignments, as long as one
  depends on the other, and doesn't redefine types.

(You'll notice by now we're in deep scary-land, trying, after compile
time, to compose type correct modules into sets of modules (packages),
and then composing those sets with each other to yield type correct
programs, some of which redefine parts of the module hierarchy, and
doing all this without implementing a type checking story for when to
combine these sets safely).

* So, the solver for cabal-install has to be updated to allow the same 
  package to have multiple, conflicting versions, as long as version X
  depends on version Y, and then not reject programs that produce this
  constraint.

* This is non trivial, but think this refactoring is possible, but it is
  hard. ultimately we're still making optimistic assumptions about 
  when module sets can be combined to produce type correct programs, 
  and conservative assumptions, at the same time.

  What we need is a semantics for packages, that in turn uses a
  semantics for modules, that explains interfaces in terms of types.

  Packages of functors, containing modules, which we can truly compose
  to yield type correct programs.

* But we think the cabal-install solver will be refactorable to let a
  good number of programs work, but it'll take a few days of constraint solver hacking.

* The end result is that cabal-install should be able to find automated
  install plans for packages that ask for base-3, even when base-4 is on
  the system as well, and it uses pieces of base-3 libraries and base-4
  libraries. Some more programs will work than if we didn't ship base-3.

* The two other ways of combining packages into Haskell programs,

    + ghc --make 
    + runhaskell Setup.hs configure
 
  will always pick base-4 now, so you won't be able to build programs
  against the base-3 without manually overriding. This means things that
  need the old exception API, for example, or syb, will break without
  manual hiding (--package base-3.0.0.0).

* We're missing hard stats on what number of things break under which
  scheme.

-- Don