[GHC] #16199: GHC 8.6 bundles libraries that don't correspond to Hackage releases

[GHC]

#16199: GHC 8.6 bundles libraries that don't correspond to Hackage releases
-------------------------------------+-------------------------------------
        Reporter:  RyanGlScott       |                Owner:  (none)
            Type:  bug               |               Status:  patch
        Priority:  highest           |            Milestone:
       Component:  libraries         |              Version:  8.6.3
  (other)                            |
      Resolution:                    |             Keywords:
Operating System:  Unknown/Multiple  |         Architecture:
                                     |  Unknown/Multiple
 Type of failure:  None/Unknown      |            Test Case:
      Blocked By:                    |             Blocking:
 Related Tickets:  #14678            |  Differential Rev(s):
       Wiki Page:                    |  https://gitlab.haskell.org/ghc/ghc/merge_requests/139
-------------------------------------+-------------------------------------

Comment (by hvr):

 Replying to [comment:5 angerman]:
 > For each of the (external) libraries, we'll run `cabal new-sdist` (1)
 > Then we'll pull the same one from hackage with `cabal get --pristine`.
 (2)
 > And finally run a diff over the unpacked one from (1) and (2).
 >
 > This should yield an empty set.

 Fwiw, we can't guarantee this in general, nor do we need to.

 There is no requirement that you can actually reinstall a boot package
 release bundled with GHC. There's no need for that, and there's packages
 that inevitably fall into this category. So tooling needs to be able to
 cope with that anyway; and in fact Cabal does.

 That being said, the only invariant that actually matters and the only one
 I care about very strongly is that of **consistent API versioning**:

 **If a package `foo-1.2.3` is bundled with GHC then if-and-only-if there
 is a valid build-plan (meeting all involved version constraints), then the
 `foo-1.2.3` from Hackage re-synthesized via Cabal must not to be
 distinguishable at the API level  from the GHC pkg-db pre-installed one.
 In other words, `foo-1.2.3` shall uniquely determine the observable API
 exposed of that package.**

 A very prominent example was the `mtl` package that shipped with GHC 8.4.1
 which broke the API versioning and caused real-world problems which (see
 https://github.com/haskell-hvr/HsYAML/issues/1 which required some very
 awkward CPP -- but in general this kind of problem is not always
 workaroundable by CPP! We ''must'' be able to trust the API versioning or
 we'd have to give up on version numbers altogether)


 Anything else, like diverging testsuites/benchmark components or other
 deltas which have no effect on the exposed API are of very little concern
 to me.

 Avoiding divergences would require enough time between the final release
 and the preceding release of a GHC last-beta or release-candidate after
 which boot package maintainers can be encouraged to publish their package
 release candidates on Hackage to the primary index (and I have to stress:
 "and no sooner!", as it's strongly discouraged to make releases on Hackage
 which target or advertise support for unreleased dependencies or tooling,
 or causing avoidable uploads to Hackage as is clearly specified in the
 Hackage upload guidelines), after we have ensured (in topological dep
 ordering) that the package candidates are sound and in sync with the
 artifacts included in the GHC release, and also fix any minor and not so
 minor issues detected in the process.

 or spelled out differently, the protocol for finalising boot libraries
 would be more or less like

 1. release a last-beta/release-candidate
 2. go over all boot libraries in topological sorted order and ensure their
 exposed API matched the Hackage release candidate; ensure that the
 compatibility advertised via version constraints is accurate; if needed,
 coordinate fixups with the maintainer
 3. restart 2. in case there's not-so-minor changes which might cause an
 avalanche effect on their rev-deps
 4. if there were fixups needed, make sure to `./validate` again;
 5. when it's clear that the release-candidate has no more issues
 (otherwise go back to 1.) and we're going to cut the final release;
 encourage package maintainers to publish their latest Hackage package
 release candidate (which we verified to match the one contained in GHC's
 release) to the main index (i.e. press that "publish" button in the
 Hackage release candidate UI)

-- 
Ticket URL: <http://ghc.haskell.org/trac/ghc/ticket/16199#comment:9>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler