[Haskell-cafe] Hackage inconsistent? (pandoc.cabal in 1.13.1)

[Mateusz Kowalczyk]

On 10/27/2014 07:48 PM, Erik Hesselink wrote:
> I was a bit hesitant about using this feature as well. We've had some
> confusion when we used it [1][2][3]. The Nix guys also seem to be
> having some trouble fitting it into their workflow [4]. But in [1] I
> did a quick grep to see how many packages used the feature, and there
> are already quite a lot. There is an 'x-revision' property added to
> the cabal file in the index, so in theory I guess tools could learn to
> deal with it.
> 
> In general I think the biggest win for this feature is not even in
> relaxing bounds, but in tightening them when it turns out they're too
> loose. Previously cabal was always free to pick an old, unconstrained
> version and would often do so, leading to type errors during builds.
> 
> Erik
> 
> [1] https://github.com/silkapp/rest/issues/57
> [2] https://github.com/silkapp/aeson-utils/issues/1
> [3] https://github.com/silkapp/json-schema/issues/12
> [4] https://github.com/NixOS/cabal2nix/issues/84
> 
> On Mon, Oct 27, 2014 at 8:30 PM, John MacFarlane <jgm at berkeley.edu> wrote:
>> I must say, I did pause a bit before making this change to
>> pandoc on Hackage, precisely because I realized that the
>> cabal file in the package would not match the one on
>> Hackage.
>>
>> But I went ahead, assuming that if the means were provided
>> to make this kind of change, it should be okay.
>>
>> It is time-consuming for me to make a proper pandoc
>> release.  I need to write up release notes, rebuild the
>> website, generate and test the binary packages on various
>> different platforms, upload to github releases and Hackage,
>> send announcement emails, etc.  Even with automation this
>> all takes work.  I'd rather not have to do this every time a
>> version bound changes on a dependent package, so I like this
>> way of making minor tweaks to version bounds; I hope I
>> can be persuaded that it's not a bad idea.
>>
>> John
>>

TL;DR version of below: I think Hackage should generate a new package
version for each info change rather than the change being ‘silent’
making it pretty exclusive to tools which don't care about what they
actually build and what hash it has (cabal). This brings us back to ‘all
version are explicit’ from the ‘we have versions but there are some
updates you need to pull in your tools too’.

Long version:

As Erik points out at his [4], this really messes things up for us in
the nix world, and really anywhere that

1. Wants to grab from Hackage
2. Wants a reproducible build

The common justification for this is that it doesn't change the tarball
hash. Well then, this ‘update info after upload’ feature then
automatically excludes every single system which uses this tarball. Why?

1. Maintainer uploads a package
2. Distros package it and note down the hash
3. Something happens and maintainer bumps the bounds on Hackage only
4. Distros still grab the very same package as they did before (tarball
doesn't change, remember?). Inside is the cabal file with old
dependencies, so the build fails if the distro updated other packages
(almost certainly).

So even though the hash doesn't change, builds suddenly start failing.
If you're using cabal, you don't get a reproducible build at all because
which versions it picks depends on whatever info is on Hackage at build
time (well, last cabal update time) and not what's in the tarball. It is
currently perfectly possible for two separate users to have ‘cabal
update’ and know about exactly the same package versions available but
end up with different version numbers used because one ‘cabal update’d
before the info on Hackage was changed and the other did not.

But John is right that sometimes making releases is tiresome. For
version bumps it can be as easy as checking out the last tag, bumping
deps and uploading, but it can be much more involved too. I admit that I
used this myself, although only for the stuff that hope to release
properly soon and I happen to maintain on NixOS too which AFAICT is the
only distro which packages Haskell fast enough to notice breakage within
few days (in huge part thanks to Peter Simons ;) )

So clearly there is a problem here: only cabal users use the information
from Hackage info bumps. Everyone else who looks into the tarball just
fails to build. cabal users who happen to cabal update at unfortunate
time also fail to build.

I can only think of one ‘solution’ which fixes this and doesn't require
considerable effort and co-ordination between maintainers and packagers.
Here is how situation currently looks like

1. I upload awesomepackage (ap) to Hackage under ap-1.0.0.0
2. Time passes, I have commits on top
3. One of those packager type of folk who like to demand things come
onto my issue tracker and say I should update my dependencies but I
don't want to release right now
4. I go onto Hackage and use that new cool feature, changing the cabal
file on ap-1.0.0.0.
5. Packager is unhappy because the problem is not fixed, ap-1.0.0.0
gives them the same thing. The only thing they can do is monkeypatch
deps on their side.

Instead, I think at steps 4 and 5 the following should happen:

4. I make changes to ap-1.0.0.0, Hackage generates ap-1.0.0.0-fix1
package with identical content but new info without any extra effort on
my side.
5. Packager simply now points to ap-1.0.0.0-fix1 which has the new hash
and builds great again. cabal does The Right Thing™ here and resolves
ap-1.0.0.0 to ap-1.0.0.0-fix1 so everything keeps working there as it
does now.

It does seem a bit wasteful space-wise but with clever programming on
Hackage side that could probably be avoided. There is no longer a
problem with these ‘hidden’ changes happening. Everyone can point at the
*exact* version they want used like they do now and they get what
exactly what they asked for. A less wasteful way would be for the
packaging distros to update their tools to look at the Hackage-only
cabal files as well. Personally I think that seems like the wrong place
to solve this though.

Lastly, Hackage should announce with big fat red letters on the front
page of the package that the source tarball contains different info than
what's shown on Hackage. It's really not possible to tell as a user at
the moment unless we specifically look for it.

Sorry for the long-winded message, hopefully you heeded the TL;DR at the
top and skipped this.

-- 
Mateusz K.