[Haskell-cafe] A new cabal odissey: cabal-1.8 breaking its own neck by updating its dependencies

[Paolo Giarrusso]

Hi!

First, sorry for some confusion - I wanted to post further details
needed for the analysis in the mail I lost, and I had excluded them
from my summary to keep it short.

On Sun, Sep 12, 2010 at 15:26, Tillmann Rendel
<rendel at mathematik.uni-marburg.de> wrote:
> Hi Paolo,

> Paolo Giarrusso wrote:

>> - when recompiling a package with ABI changes, does cabal always
>> update dependent packages?

> It never recompiles them. Recompilation should not be needed, because
> different versions of packages exports different symbols, so a package can
> never be linked against the wrong version of its dependency.

My problem was with reinstalling the same version of a package. Again,
sorry for the confusion (see above). cabal should then recompile (not
"update", strictly speaking) dependent packages:

Recompiling the same version of a package does not always yield the
same result, ABI-wise.

The problem was an ABI difference between two compilation results for
old-time-1.0.0.2, one linked against old-locale-1.0.0.1, the other
against version 1.0.0.2 of the same package. The ABI difference was
_probably_ due to different cross-module inlining decisions: the
disappeared symbols were:

oldzmtimezm1zi0zi0zi2_SystemziTime_a149_{closure,info},

i.e. (I assume) old-time-1.0.0.2-System.Time.a149. That name is
generated by GHC during optimization, and my educated guess is that it
is exported to allow cross-module inlining. In particular, a149 is
mentioned in the .hi interface of old-time's System.Time - <libs
dir>/old-time-1.0.0.2/System/Time.hi

It thus _seems_ that the ABI of a module is a (pure) function of the
versions of all its (transitive) dependencies, and their compilation
options - in particular, the optimization options and the used
compiler version.

More formally, my conjecture is:
- let us represent "package a depends on b" as "a =>> b", where a, b
are package names tagged with a version
- let =>>* be the transitive-reflexive closure of =>>
we need to compute:
DEPS(p) = {q | p =>>* q }
TAGGED_DEPS(p) = { (q, compilation_opts(q)) | q \in DEPS(p) }
where compilation_opts(q) are the compilation options used to compile package q.

Then, the ABI of a module is a pure function of TAGGED_DEPS(p), not
just of p. My experience proves at least that the ABI does not depend
just on p.

And since, after the discussion up-to-now, it turns out that this is
unexpected, I reported this as a bug:
http://hackage.haskell.org/trac/hackage/ticket/738

Another result of the above is that a mere "cabal install --reinstall
-O2 FooPackage", recompiling the same version of FooPackage with -O2
instead of the default (-O1), requires recompiling all packages which
depends on (i.e. use symbols from) FooPackage; recompiling them
implies recompiling packages depending on them, and so on, in a
"transitively closed" way.

<BINARY_LIBRARIES>
A final result is that shipping Haskell libraries in binary form (i.e.
closed source libraries) is not supported without major changes. Not
that I care directly, but somebody might, especially since you choose
the BSD license over the GPL one. And fixing that might also make
package upgrades more "sustainable", i.e. allowing to upgrade a
library while rebuilding less dependent modules. To ship a binary
library, one would need:

- disabling cross-module inlining when building the closed source
library, but that's not enough: one still needs to link to a specific
version of a package, because the version number is mangled into the
name, up to the 4th version level. That's inconvenient, because a
library can't benefit from a change affecting only the implementation,
and not the interface, of a library (e.g., a bugfix or a security
fix).

== On .NET, changes to the last version number must be
binary-compatible, so 1.0.0.1 and 1.0.0.2 are equivalent at link time,
and 1.0.0 and 1.0.1 are incompatible but can coexist.

== On ELF, for libraries which care (say glibc), each symbol is
available (with the same ABI) from a given library version onwards;
client binaries depend on at dynamic linking time on a library version
>= the one they needed. An "Hello World program", using no recent
functions, could thus be backportable even to an earlier version of
the C library. But I don't propose that, it requires enormous effort.
Most other libraries use a Scheme which is similar to the .NET one,
barring some big but superficial differences.

- resorting to full static linking, if that's supported; even then,
one can't statically link to the runtime system, and typechecking
needs to use versioned typenames.

</BINARY_LIBRARIES>

> However, see
> the following tickets:

>  http://hackage.haskell.org/trac/hackage/ticket/701
>  http://hackage.haskell.org/trac/hackage/ticket/704

Had a look, thanks - but they do not apply here, the problem is with
Haskell symbols.

>> Interestingly, HTTP, directory, process, zip-archive were not
>> reinstalled, which confirms that Cabal had reinstalled them before
>> just because of an upgrade to the dependencies.

> I think you are misinterpreting this.

> When you asked cabal-install to install pandoc, it tried to make a
> consistent install plan for all its transitive dependencies. cabal-install
> will not touch a package which is not a transitive dependency of the package
> you request to be installed. Therefore, cabal-install will not touch Cabal
> if you ask it to install pandoc.

Thank you very much for the explanation. Note that the policy you
describe is not necessarily "the right one", it is just a choice which
can work, with the fixes you propose (read on for alternatives).

> To make a consistent install plan, cabal-install has to pick exactly one
> version number for each of the transitive dependencies, so that all version
> constraints of all transitive dependencies are fullfilled. For some reason,
> cabal-install picked old-locale-1.0.0.2 instead of the already installed
> old-locale-1.0.0.1, and newer versions of HTTP, directory etc. too.

My case was slightly different, old-locale-1.0.0.2 (user-level) and .1
(system level) were already present.
old-time-1.0.0.2 was installed at the system level, and linked against
the older, system-level old-locale. So, while no dependency was
broken, old-locale had been upgraded, by me or by cabal on its own.
Even if it was my fault, only "cabal upgrade" gives a warning, while
"cabal install --help" doesn't mention the issue, which is
"suboptimal" :-D.

And of course, cabal should prevent a user from shooting himself in
his foot, but that's a more advanced feature.

A consistent plan for installing pandoc would have been to either
compile it using the older old-locale, or to recompile old-time
against the new version. Sadly, cabal chose the latter alternative,
since it is not obviously wrong, and this broke everything, except
packages on which pandoc depended: they were recompiled because of the
changed dep.
I later forced the other install plan, with some pain, but it works.

So cabal behavior is inconsistent: it knows that recompiling a package
means recompiling its dependencies, but it should either do so also
for packages unrelated to the one being installed, or it should not
recompile any package (as you suggest).

> I think this is the bug: cabal-install should not be allowed to install
> old-locale, because doing so apparantly causes havoc.

Lacking a proper dependency handling (which _is_ not trivial), yes.
And it's the best current choice.
But one can recompile and upgrade on a running system, without
downtime nor glitches, both the C standard library and libraries
without stable ABIs (see Gentoo Linux). Thus, upgrading old-locale and
automatically rebuilding its dependencies would be a sensible feature.
It just should not happen unless the user requests a full upgrade.

Finally, while old-locale is a core package and one can restrict its
upgrades, if I recompile non-core package A, and non-core package B
breaks, that's still "havoc" - the only difference is that B is not
cabal, and the bug is thus less severe and you can recompile manually
B.
OTOH, fixing this less severe and more complicated bug, via
transitively closed dependency tracking as described above, helps
supporting complete upgrades.

> Looking at the inter-dependencies of pandoc's transitive dependencies, I do
> not see a reason to install a new version of a package instead of keeping
> the old.

> Maybe it's somehow related to the transition from base-3 to base-4?
The newer old-locale was already installed - see above my description
of the two possible install plans.

For the transition, I had added:
preference: base >= 4
based on a suggestion from somebody on haskell-cafe and the resulting
discussion.
-- 
Paolo Giarrusso - Ph.D. Student
http://www.informatik.uni-marburg.de/~pgiarrusso/