[GHC DevOps Group] Release policies

[Simon Peyton Jones]

This thread sounds as if it has been productive, though I have not followed the details.

Does anyone feel able to draw it together into a proposed release policy?  Along with a summary of the reasoning that led to it?

Thanks

Simon

|  -----Original Message-----
|  From: Ghc-devops-group [mailto:ghc-devops-group-bounces at haskell.org] On
|  Behalf Of Boespflug, Mathieu
|  Sent: 18 December 2017 11:03
|  To: Manuel M T Chakravarty <manuel.chakravarty at tweag.io>
|  Cc: ghc-devops-group at haskell.org; ghc-devs at haskell.org Devs <ghc-
|  devs at haskell.org>
|  Subject: Re: [GHC DevOps Group] Release policies
|  
|  I think Mikhail's point is that if a package says build-type: Simple, then
|  we know exactly what its Setup.hs says, and therefore also which part of the
|  Cabal API it's using. Easy enough to keep that part stable even if others
|  change. Case in point: Cabal-2.0 brought a number of changes to the overall
|  API, but nothing that broke calling defaultMain from Distribution.Simple
|  (which is what a build-type: Simple script does). At the end of the day the
|  consumers of the wider Cabal API are pretty small. A substantial number of
|  misc packages on Hackage do it but rarely heavily. Other than that it
|  essentially comes down to Stack, cabal-install, Haskell For Mac and... any
|  others?
|  
|  My takeaway from the discussion so far is that the number of heavy consumers
|  looks small enough that a draconian BC policy for Cabal-the-library sounds
|  overkill, provided, crucially, that everything is in place, by GHC feature
|  freeze at the very latest, to allow a smooth migration. A "smooth
|  transition" means having a migration guide available before start of feature
|  freeze etc, but to Ben's concern stated earlier in this thread (about
|  GHC/upstream coupling), ideally also a release.
|  
|  I should note that to the extent that GHC tracks upstream releases only (not
|  git commits in unknown state), GHC can be released on a timely schedule
|  without needing any coordination from upstream maintainers to await new
|  releases on their part. So quite apart from the Cabal thing specifically,
|  it's worth thinking about asking that the versions of all upstream packages
|  only make it into GHC, at the behest of their respective maintainers, after
|  a new release of upstream is made. This was already proposed earlier in the
|  thread:
|  
|  > * [Proposal:] GHC does not track git commits of upstream dependencies
|  > in an unknown state of quality, but tracks vetted and tested releases
|  > instead.
|  
|  Potentially, this could even mean drastically cutting down on the number of
|  git submodules carried in the GHC repo. Since these packages could as well
|  be downloaded from Hackage.
|  
|  
|  On 18 December 2017 at 05:04, Manuel M T Chakravarty
|  <manuel.chakravarty at tweag.io> wrote:
|  > Yes, you are right Haskell for Mac also links against Cabal-the-library
|  and API changes have regularly required me to fix my code. I guess, I have
|  never been particularly stressed about it, because I also link against GHC
|  API and that doesn’t even know how to spell API stability — i.e., changes
|  required by Cabal are usually drowned out by the chaos inflicted by GHC.
|  >
|  > In any case, you are making a good point.
|  >
|  > Mikhail, I don’t understand your response to Mathieu at all. What does the
|  build-type have to do with this?
|  >
|  > Cheers,
|  > Manuel
|  >
|  >> 15.12.2017, 19:41 Boespflug, Mathieu <m at tweag.io>:
|  >>
|  >> Thanks for the feedback, Michael.
|  >>
|  >> Manuel, I believe you are also a Cabal-the-library consumer in Haskell
|  For Mac?
|  >>
|  >> Michael, you brought up another problem tangentially related to the
|  >> original integer-gmp issue but that was not in my original list
|  >> earlier in this thread:
|  >>
|  >> * Cabal-2.0.0 had breaking changes in the API.
|  >>
|  >> This means that by association GHC itself broke BC, because it
|  >> shipped with Cabal-2.0, without the usual grace period.
|  >>
|  >> Now, there are far fewer users of Cabal than of base. All, Michael in
|  >> his previous email seems to be okay with breaking changes in Cabal
|  >> given the conditions he stated (2 months grace period, advance notice
|  >> of when the 2 months start). And perhaps this points to the lack of a
|  >> need for the regular grace period applying to Cabal. How many other
|  >> users of Cabal-the-library are there? In principle, every single
|  >> Hackage package out there, which all have a Setup.hs script. Most of
|  >> them are trivial, but how many did break because of these API changes?
|  >> I for one am pretty happy for Cabal to move fast, but I'm concerned
|  >> that these breaking changes happened without any kind of advance
|  >> notice. To Simon's original point - there does not to be a clear
|  >> policy and a good process surrounding Cabal itself and other GHC
|  >> dependencies. So far we discussed mostly metadata changes, not API
|  >> changes.
|  >>
|  >> And to be clear, folks did get some (post facto) notice in September:
|  >> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fcoldw
|  >> a.st%2Fe%2Fblog%2F2017-09-09-Cabal-2-0.html&data=02%7C01%7Csimonpj%40
|  >> microsoft.com%7Ca4ef8935c10c4da1766f08d546070073%7C72f988bf86f141af91
|  >> ab2d7cd011db47%7C1%7C0%7C636491918245390697&sdata=YNAuG7XcS9NiRMyklKe
|  >> 1LyL6LrxKr%2BlIsu6oxFXb%2Byg%3D&reserved=0. That's helpful, but I
|  >> submit that in the future this really should be part of the GHC release
|  announcement (which happened over a month before that), and in fact a
|  migration guide circulated before the feature freeze, so downstream tooling
|  authors can adapt. If this is not possible, then perhaps it's premature for
|  GHC to include that given Cabal release.
|  >> Again, GHC should always have the option to stick to the old Cabal
|  >> version until things get ironed out.
|  >>
|  >>
|  >> On 15 December 2017 at 08:42, Michael Snoyman <michael at snoyman.com>
|  wrote:
|  >>>
|  >>>
|  >>> On Thu, Dec 14, 2017 at 12:27 PM, Boespflug, Mathieu <m at tweag.io> wrote:
|  >>>
|  >>> [snip]
|  >>>
|  >>>> * Or a middle ground: make feature freeze a thing. Meaning that for
|  >>>> a couple of months before a major GHC release, the major new Cabal
|  >>>> isn't technically released yet, but like GHC itself within this
|  >>>> period, it's pretty staid, so not so much a moving target, and
|  >>>> something downstream tooling authors can possibly adapt to even
|  >>>> without any grace period on new metadata features. This assumes
|  >>>> that the 2 months of feature freeze are enough time for downstream
|  >>>> tooling. Thoughts from any of those maintainers?
|  >>>>
|  >>>
|  >>> Short answer: if there's a clear idea in advance of when this
|  >>> feature freeze is going to happen, I think we can coordinate
|  >>> releases of downstream tooling (Stack being the most important, but
|  >>> stackage-curator playing in as well) so that 2 months is sufficient.
|  >>> I'll talk with the rest of the Stack team to see if there are any
|  concerns.
|  >>>
|  >>> Longer answer: Stack intentionally avoids depending on the internals
|  >>> of Cabal wherever possible. Instead of calling library functions
|  >>> directly from within Haskell code to perform builds, for example, it
|  >>> interacts with the Setup.hs files over their command line interface.[1]
|  This has two results:
|  >>>
|  >>> * Stack can usually start using new GHC/Cabal versions without a new
|  >>> Stack release, since it's just shelling out for the actual build
|  >>> * There's not usually very much code churn needed in Stack to
|  >>> upgrade to a newer Cabal release
|  >>>
|  >>> This past release was an exception because of all of the changes
|  >>> that landed, both the new cabal grammar to support the ^>= operator
|  >>> (making the old parser incapable of lossily parsing new files) and
|  >>> API changes (I think mostly around Backpack, though there was some
|  >>> code cleanup as well). In particular, the main interface we need
|  >>> from Cabal—the package description data types and parser—changed
|  >>> significantly enough that it took significant effort to upgrade.
|  >>> There were also new features added (like sub libraries and foreign
|  >>> libraries) that weren't immediately supported by the old Stack version,
|  and had to be manually added in.
|  >>>
|  >>> Tying this up: generally upgrading to a new Cabal release should be
|  >>> fine, and the only concern I'd have is fitting it into a release
|  >>> schedule with Stack. The complications that could slow that down are:
|  >>>
|  >>> * Changes to the command line interface that Stack uses (hopefully
|  >>> those are exceedingly rare)
|  >>> * Major overhauls to the Stack-facing API
|  >>>
|  >>> Michael
|  >>>
|  >>> [1] This allows for more reproducible builds of older snapshots,
|  >>> insuring that the exact same Cabal library is performing the builds
|  >> _______________________________________________
|  >> Ghc-devops-group mailing list
|  >> Ghc-devops-group at haskell.org
|  >> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group
|  >
|  _______________________________________________
|  Ghc-devops-group mailing list
|  Ghc-devops-group at haskell.org
|  https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group