[GHC DevOps Group] Release policies

Mon Dec 18 11:18:26 UTC 2017

Hi,

this might only be tangentially relevant.  However, you might consider
this a working example of GHC and Cabal bleeding edge symbiosis.

Some might have seen that I built some *relocatable* GHC releases for
cross compilation (however those include the full base compiler for macOS
and linux (deb8) as well) at http://hackage.mobilehaskell.org/.

To facility those builds, I did not only need to make changes to GHC
to allow it to detect its library folder relative to `ghc`.  I also did
made quite a lot of changes to hadrian, and cabal(!) to actually support
describing GHC packages (primarily rts and ghc-prim) as cabal packages
without resorting to hacks in ghc-cabal. And getting rid of ghc-cabal
altogether.

Any overly restrictive policy on GHC and Cabal releases will result in
changes like theses to become a lot harder to do.  After all a lot of
work on GHC is performed by volunteers in their spare time, and I
honestly can not see myself forcing someone else to spend even more time
on streamlining project integrations, if even higher barriers are put
into place.  If we start dropping submodule dependencies, we need to make
it trivial to put them back into place when hacking on GHC.

I would therefore rather suggest we keep submodules in the development
tree of GHC, but put a policy into place that says that submodules have
to be replaced with their proper hackage dependencies (and versions)
prior to any release.  This would allow to keep working across ghc and
multiple dependencies as source, while giving some additional non-in-flight
guarantees about GHC releases and their dependencies.

Another example would be my llvm-ng (bitcode producing llvm backend),
which is naturally also a set of submodules in my ghc tree, as working on
the code gen and ghc at the same time would either mean I need to fully
integrate those libraries into GHC, or start bumping them on hackage for
every minimal change while working on GHC and the code gen.

By the end of the day, I believe we want to make hacking on GHC as easy
for everyone as possible while at the same time improving the release
quality.

Cheers,
 Moritz

> On Dec 18, 2017, at 7:03 PM, Boespflug, Mathieu <m at tweag.io> wrote:
> 
> I think Mikhail's point is that if a package says build-type: Simple,
> then we know exactly what its Setup.hs says, and therefore also which
> part of the Cabal API it's using. Easy enough to keep that part stable
> even if others change. Case in point: Cabal-2.0 brought a number of
> changes to the overall API, but nothing that broke calling defaultMain
> from Distribution.Simple (which is what a build-type: Simple script
> does). At the end of the day the consumers of the wider Cabal API are
> pretty small. A substantial number of misc packages on Hackage do it
> but rarely heavily. Other than that it essentially comes down to
> Stack, cabal-install, Haskell For Mac and... any others?
> 
> My takeaway from the discussion so far is that the number of heavy
> consumers looks small enough that a draconian BC policy for
> Cabal-the-library sounds overkill, provided, crucially, that
> everything is in place, by GHC feature freeze at the very latest, to
> allow a smooth migration. A "smooth transition" means having a
> migration guide available before start of feature freeze etc, but to
> Ben's concern stated earlier in this thread (about GHC/upstream
> coupling), ideally also a release.
> 
> I should note that to the extent that GHC tracks upstream releases
> only (not git commits in unknown state), GHC can be released on a
> timely schedule without needing any coordination from upstream
> maintainers to await new releases on their part. So quite apart from
> the Cabal thing specifically, it's worth thinking about asking that
> the versions of all upstream packages only make it into GHC, at the
> behest of their respective maintainers, after a new release of
> upstream is made. This was already proposed earlier in the thread:
> 
>> * [Proposal:] GHC does not track git commits of upstream dependencies
>> in an unknown state of quality, but tracks vetted and tested releases
>> instead.
> 
> Potentially, this could even mean drastically cutting down on the
> number of git submodules carried in the GHC repo. Since these packages
> could as well be downloaded from Hackage.
> 
> 
> On 18 December 2017 at 05:04, Manuel M T Chakravarty
> <manuel.chakravarty at tweag.io> wrote:
>> Yes, you are right Haskell for Mac also links against Cabal-the-library and API changes have regularly required me to fix my code. I guess, I have never been particularly stressed about it, because I also link against GHC API and that doesn’t even know how to spell API stability — i.e., changes required by Cabal are usually drowned out by the chaos inflicted by GHC.
>> 
>> In any case, you are making a good point.
>> 
>> Mikhail, I don’t understand your response to Mathieu at all. What does the build-type have to do with this?
>> 
>> Cheers,
>> Manuel
>> 
>>> 15.12.2017, 19:41 Boespflug, Mathieu <m at tweag.io>:
>>> 
>>> Thanks for the feedback, Michael.
>>> 
>>> Manuel, I believe you are also a Cabal-the-library consumer in Haskell For Mac?
>>> 
>>> Michael, you brought up another problem tangentially related to the
>>> original integer-gmp issue but that was not in my original list
>>> earlier in this thread:
>>> 
>>> * Cabal-2.0.0 had breaking changes in the API.
>>> 
>>> This means that by association GHC itself broke BC, because it shipped
>>> with Cabal-2.0, without the usual grace period.
>>> 
>>> Now, there are far fewer users of Cabal than of base. All, Michael in
>>> his previous email seems to be okay with breaking changes in Cabal
>>> given the conditions he stated (2 months grace period, advance notice
>>> of when the 2 months start). And perhaps this points to the lack of a
>>> need for the regular grace period applying to Cabal. How many other
>>> users of Cabal-the-library are there? In principle, every single
>>> Hackage package out there, which all have a Setup.hs script. Most of
>>> them are trivial, but how many did break because of these API changes?
>>> I for one am pretty happy for Cabal to move fast, but I'm concerned
>>> that these breaking changes happened without any kind of advance
>>> notice. To Simon's original point - there does not to be a clear
>>> policy and a good process surrounding Cabal itself and other GHC
>>> dependencies. So far we discussed mostly metadata changes, not API
>>> changes.
>>> 
>>> And to be clear, folks did get some (post facto) notice in September:
>>> http://coldwa.st/e/blog/2017-09-09-Cabal-2-0.html. That's helpful, but
>>> I submit that in the future this really should be part of the GHC
>>> release announcement (which happened over a month before that), and in
>>> fact a migration guide circulated before the feature freeze, so
>>> downstream tooling authors can adapt. If this is not possible, then
>>> perhaps it's premature for GHC to include that given Cabal release.
>>> Again, GHC should always have the option to stick to the old Cabal
>>> version until things get ironed out.
>>> 
>>> 
>>> On 15 December 2017 at 08:42, Michael Snoyman <michael at snoyman.com> wrote:
>>>> 
>>>> 
>>>> On Thu, Dec 14, 2017 at 12:27 PM, Boespflug, Mathieu <m at tweag.io> wrote:
>>>> 
>>>> [snip]
>>>> 
>>>>> * Or a middle ground: make feature freeze a thing. Meaning that for a
>>>>> couple of months before a major GHC release, the major new Cabal isn't
>>>>> technically released yet, but like GHC itself within this period, it's
>>>>> pretty staid, so not so much a moving target, and something downstream
>>>>> tooling authors can possibly adapt to even without any grace period on
>>>>> new metadata features. This assumes that the 2 months of feature
>>>>> freeze are enough time for downstream tooling. Thoughts from any of
>>>>> those maintainers?
>>>>> 
>>>> 
>>>> Short answer: if there's a clear idea in advance of when this feature freeze
>>>> is going to happen, I think we can coordinate releases of downstream tooling
>>>> (Stack being the most important, but stackage-curator playing in as well) so
>>>> that 2 months is sufficient. I'll talk with the rest of the Stack team to
>>>> see if there are any concerns.
>>>> 
>>>> Longer answer: Stack intentionally avoids depending on the internals of
>>>> Cabal wherever possible. Instead of calling library functions directly from
>>>> within Haskell code to perform builds, for example, it interacts with the
>>>> Setup.hs files over their command line interface.[1] This has two results:
>>>> 
>>>> * Stack can usually start using new GHC/Cabal versions without a new Stack
>>>> release, since it's just shelling out for the actual build
>>>> * There's not usually very much code churn needed in Stack to upgrade to a
>>>> newer Cabal release
>>>> 
>>>> This past release was an exception because of all of the changes that
>>>> landed, both the new cabal grammar to support the ^>= operator (making the
>>>> old parser incapable of lossily parsing new files) and API changes (I think
>>>> mostly around Backpack, though there was some code cleanup as well). In
>>>> particular, the main interface we need from Cabal—the package description
>>>> data types and parser—changed significantly enough that it took significant
>>>> effort to upgrade. There were also new features added (like sub libraries
>>>> and foreign libraries) that weren't immediately supported by the old Stack
>>>> version, and had to be manually added in.
>>>> 
>>>> Tying this up: generally upgrading to a new Cabal release should be fine,
>>>> and the only concern I'd have is fitting it into a release schedule with
>>>> Stack. The complications that could slow that down are:
>>>> 
>>>> * Changes to the command line interface that Stack uses (hopefully those are
>>>> exceedingly rare)
>>>> * Major overhauls to the Stack-facing API
>>>> 
>>>> Michael
>>>> 
>>>> [1] This allows for more reproducible builds of older snapshots, insuring
>>>> that the exact same Cabal library is performing the builds
>>> _______________________________________________
>>> Ghc-devops-group mailing list
>>> Ghc-devops-group at haskell.org
>>> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group
>> 
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs