cabal for ghc 6.8.2, package upgrade warnings, please?

[Duncan Coutts]

On Sat, 2007-12-01 at 14:05 +0000, Claus Reinke wrote:
> > We're pretty much fully backwards compatible with old
> > .cabal files, the only problem is Setup.hs files.
> 
> compatible, as in "can process them", but some things,
> like the base split, will lead to plain failures, even though
> cabal could put 2 and 2 together when it sees an old
> .cabal file and missing dependencies in packages split
> from base.
> 
> the package maintainer hasn't updated, the package
> builder has no idea what goes wrong, the whole thing
> breaks because of outside influences (libraries updates),
> and cabal could make a good guess, it just doesn't tell.

I'm not sure we actually can guess if a package is broken by the base
split or not.

build-depends: base

so does it really only depend on base or does it depend on base, pretty,
directory etc etc. No idea.

We can take secondary things into account and note the lack of newer
fields and syntax as an indication of an older package.

> > This would be great, unfortunately I don't think it's technically
> > feasible with the current design. At the moment, the Setup.hs is the
> > first entry point and if that doesn't compile then we're stuffed. We
> > have no opportunity to offer helpful messages, the user just gets a
> > deeply unfriendly type error message.
> 
> ah, yes, that is a problem. but even if non-standard Setup.hs
> is now discouraged, it is still part of cabal's design, so the same
> issue will come up the next time you improve the api.
> 
> what about the old idea that there should be a 'cabal' script?
> in the past, that seemed somewhat superfluous, as it wouldn't
> have done anything but call "runhaskell Setup", but now it
> seems that extra level of indirection would allow for helpful
> checks and messages before or after compilation of Setup.hs
> fails.

We do have this now and we'll be promoting it more in future. We
currently have the cabal-setup program and the cabal-install program.
We're currently working on merging the two into a more general cabal
front end / cli UI program.

cabal-setup does some of these things, it reads the .cabal file first
and then if necessary it compiles the Setup.hs script against some
hopefully appropriate version of the Cabal library. Unfortunately that
doesn't help much since packages generally only specify lower not upper
bounds on cabal version numbers:

cabal-version: >=1.2

and the like. This is the same problem as we have with package
dependencies. The package version policy is one approach to solving
this.

> that could include format and age checks on .cabal (if
> it doesn't mention any new dependencies or isn't precise in
> versions, there are some likely issues it will encounter with
> a new ghc/cabal version; if it depends on base, but needs
> containers, it is pre-base-split, etc.), import/type checks
> on Setup.hs (does it expect the old or the new api), or
> rough postprocessing of error messages from compiling
> Setup.hs (pattern matching on the kind of type errors or
> missing import errors that unambiguously imply old api).

We currently cannot check if a package depends on containers without
declaring it because Cabal does not do any of it's own module chasing.
When it does we'll be able to do a bit better here.

> > Setup.hs files are allowed to use the entire Cabal api, which includes a
> > lot of useful stuff but it completely exposes the inner workings. Since
> > we keep improving the inner workings that api keeps changing. Even the
> > more public parts place constraints on the implementation that we would
> > like to lift in future (esp the UserHooks api).
> > ..
> 
> ok, i think i've accepted before that cabal's design would
> make it difficult to have a compatibility module. and you
> say that this design gives you liberties that you wouldn't
> want to trade for a more restrictive, more maintenance-
> friendly design. okay, but that isn't the only way to help
> with maintenance:
> 
> - prefixing 'Compat.' to all cabal imports in Setup.hs
>     seems out - too much duplication of implementation.
>     but perhaps that duplication of implementation would
>     have been worth it for a few months, if only to be
>     able to issue warnings and give package authors or
>     third parties some time to upgrade their packages.

It's not the biggest problem. As I said it only broke 10% of packages,
the vast majority of packages were broken by the base split.

> - if the new api can do everything the old api can do,
>     is it possible to transform an old Setup.hs into one
>     that would very nearly work with the new api?
> 
>     using Language.Haskell's parser/ast/pretty-printer
>     and one of the no-boilerplater libraries, one could
>     search for uses of the old api and replace with
>     uses of the new api wherever drop-in replacements
>     are available. the bulk of the code would be rules
>     encoding old->new replacements.

Sounds like a lot of work for relatively little benefit.

>     when you changed cabal's api, you must have had
>     some ideas about how the new api would replace
>     uses of the old api?

Some, but as I've said, there's no clear line about what is a public
Cabal api and what is just internal stuff. We currently export pretty
much everything. Most of the api bits we change we do not expect many
packages are using. We do try rather harder not to change the bits we
expect more packages use. Generally we consider the internal/external
line to be Distribution.Simple.* vs Distribution.* but there are
certainly various useful things that some Setup.hs scripts use from
within Distribution.Simple. We don't have a very good feel for what
those bits are exactly.

> - even if the transformation cannot be automated
>     completely, can the old code be analysed to
>     issue warnings about features it uses which are
>     no longer available, together with a link to a wiki
>     page that explains how to replace those uses of
>     old cabal api features with uses of new cabal
>     api features? again, give the package builder
>     more info than "it doesn't compile/build".
> 
> - is there a wiki page that explains the cabal api
>     changes in more detail, with rationale, with
>     examples of how to replace old uses with new?
> 
>     something that cabal and hackage home pages,
>     as well as warning issued by cabal, could point to.

This is all stuff we should have of course. My immediate reaction I
admit is that it sounds like a lot of work that might be better invested
in just improving the design of cabal generally.

> > The current approach works about 90%, in that 10% of packages broke in
> > the transition from Cabal 1.1.6 to 1.2 (according to my hackage survey).
> 
> that survey was useful, but it only covered hackage,
> and even there it found 10% breakage not caused
> by package authors or builders. 10% breakage
> just by time passing and by cabal not wanting to
> provide the old implementation side-by-side with
> the new, not by package providers making any changes..

We can and do provide the old implementation side-by-side. You can
install multiple versions of the Cabal lib.

> as you say, the cabal is going to move on, so this is
> an opportunity to see what issues come up and how
> things might be changed to handle such issues better
> next time.
> 
> even simple things help, such as an additional
> indirection (a 'cabal' script calling 'runhaskell Setup',
> now able to give warnings instead of fail), wiki pages
> documenting what people have to do to follow cabal
> or library changes, warning messages that direct
> package builders to such wiki pages.

Or designing the Cabal API with external users in mind and making
everything else private.

> is there a hackage mailing list offering to keep all
> package maintainers informed about build results,
> breakage, issues, api changes, discussion?

There is this list. The hackage build results to not get emailed to
anyone.

> the email addresses could be extracted from the .cabal files
> being uploaded, but hackage would need to ask
> for permission before adding anyone to such list.
> 
> > I've added a link from the cabal website and I've reorganised the page
> > so it's hopefully rather clearer:
> >
> > http://haskell.org/haskellwiki/Upgrading_packages
> 
> thanks. the part that still needs expansion is the part
> 1.3 on cabal api changes: any feature that the old
> cabal made available which is not unchanged in the
> new api should be listed, side-by-side with intended
> replacements.

There are a lot of api changes that are pretty much irrelevant for most
Setup.hs scripts. We could list the top few most likely changes
summarising from the hackage survey email I sent round.

> that is, a user-level equivalent to the page that would
> explain and summarize the cabal design changes.


What you're proposing generally really is a lot of work and I think
requires rather more active Cabal hackers, or a much slower rate of
change.

Duncan