Hackage is flooded with old package versions reuploads

[Herbert Valerio Riedel]

On 2015-01-19 at 00:05:31 +0100, Vincent Hanquez wrote:

[...]

>> While this is harmless I wonder what idea stands behind this?

> This is not harmless. This is a security issue by itself, as now
> packages get changes transparently given a url, you might have a
> different package one day, which trigger hash check failure. or signed
> tag verification failure.

Just to clarify: the source tarball are *never* modified to specifically
make sure hashes don't randomly change (which would trip all sorts of
warnings for distribution packagers). In fact, there's also a
`cabal get` option if you want to unpack with the original unaltered .cabal
file:

    --pristine    Unpack the original pristine tarball,
                  rather than updating the .cabal file
                  with the latest revision from the
                  package archive.

> This has also the effect of not changing the bounds in the repository,
> so for example, next time you upload a tweak'ed packages, you
> effectively revert the change done on hackage only.
> This is also done without the consent of the maintainer of a given
> package, nor that the maintainer is actually notified when that
> happens, or allow to prevent it happening.

Fwiw, there's plans for an automated notification feature:

  https://github.com/haskell/hackage-server/issues/230

We're actively working on refining the system, and finding out where it
needs to be improved. For most of my edits I did in fact inform upstream
to make sure the broken install-plan issue was known so future uploads
could address the issue (or at least add missing bounds to avoid
regressing again).

> This is pretty big start from the other similar policy for taking over
> packages, that insist on a very long period of repeated communication
> with the author and then the community.

We probably need a broader discussion to clarify what Hackage's goal
shall be in terms of (various dimensions of) stability, what policies
Hackage trustees shall follow to help achieve these goals, and most
importantly, which responsibilities a package author agrees to (and
which (s)he delegates to trustees), when (s)he uploads a package to
Hackage, which is a shared and vital resource of our ecosystem and IMO
ought to be kept in good condition to ensure a good experience for all
its users.

Right now there's varying degrees of expectations of what Hackage is,
with one extreme being that Hackage is simply considered a storage-space
for dumping package source-tarballs to (and where .cabal meta-data
hygiene has low to no priority), and the other end of the spectrum being
the vision of Hackage as a somewhat curated (and yet bleeding-edge)
package repository.

The latter is not unlike Stackage (or the LTSHaskell idea), but with a
different priority of eliminating broken install plans (while retaining
the degree of freedom version-ranges for dependencies provide) rather
than providing a single vetted cross-section of Hackage.

Even with the legitimate existence of Stackage (and possibly
LTSHaskell), there's still the need for keeping Hackage in good shape
meta-data-wise:


 1) There's a significant amount of users which are stuck with an older
    GHC (Debian Stable currently ships with GHC 7.4.2, Debian Jessie
    will most likely ship with GHC 7.6.3 later this year; I've seen
    reports of RHEL6 users being stuck with GHC 7.0.4)

    Also, the Haskell Platform lags behind one major GHC version for
    some time till it picks up the latest stable GHC version. It'll be
    quite some time till we see a HP release featuring GHC 7.10.x after
    GHC 7.10.1 is released.

    While I don't consider it paramount for every latest package version
    to still be compatible with GHC 7.4.2 (or GHC 7.0), I do consider it
    important that a `cabal install $PKG` picks the latest still
    compatible package version (if one exists). The more popular $PKG
    is, the more important I consider this.

    Just consider a new Haskell user who doesn't know all the issues
    with Hackage who reads a blog-post (I'm thinking of something like
    "24 Days of Hackage"), then goes on to `cabal install $PKG` and is
    confronted with a compile-error (rather than picking an older but
    working version)... is that the first impression we want new users
    to have with Haskell & Hackage?
    
 2) Before a new package version can be cherry-picked into
    Stackage/LTSHaskell, it needs to be on something like Hackage first,
    also to allow other packages to be updated against the new version
    (which can result in a cascade of package version updates throughout
    the dependency-graph).

    This is IMO one killer feature Hackage has with its version-ranges
    instead of single-version pinned down package-sets. I think such
    update scenarios would be quite harder to perform if Stackage didn't
    have Hackage as the staging/integration area for version-update
    waves.

 3) As an extension of 2), library maintainers also need properly
    working build-plans for older GHCs for latest-versions of the
    build-deps they rely on, as otherwise they have a hard time
    providing and ensuring backward compatibility for their packages
    during development if e.g. their Travis-job-matrices start falling
    apart due to suddenly beaking install-plans.

    The alternative would be to have package versions restrict their
    support to single GHC major versions and provide bugfixes (and
    improvements) for older GHCs via backports to previous
    package-version branches.

    Personally, I consider it less effort to have the latest
    package-version support a couple of recent GHC version, and slowly
    phase out support for older GHCs if it becomes to costly to support
    those (because e.g. features from newer GHCs are exploited)

    That results in a "diagonal support corridor" in the build-reports
    as in e.g. 

      https://ghc.haskell.org/~hvr/buildreports/either.html

    Ideally, if you can support older GHCs, then the pattern looks like
    a lower-left support triangle as in

      https://ghc.haskell.org/~hvr/buildreports/unordered-containers.html


This email's already gotten much longer than I intended to and I started
digressing... so I'm better stop here... :-)

Cheers,
  hvr