ANN: Overlay Hackage Package Index for GHC HEAD

Mon Sep 18 16:37:08 UTC 2017

Hi Herbert,

This sounds like a great idea :) Glad to see ideas from Eta inspiring new
ideas in GHC.

"constantly nagging upstream maintainers to release quickfixes
for unreleased GHC HEAD snapshots which will likely break again a few
weeks later"

^ This is what motivated us to start eta-hackage in the first place. In our
case, we didn't know whether the patches were completely correct when
porting C FFI to Java FFI, at least in the corner cases, so we needed a way
to quickly deploy updates to patches without having the end user wait a
significant amount of time. Once they stabilise, we do plan on pushing the
patches upstream to the respective packages - we've already done that for a
few of them.

Thanks,
Rahul

On Mon, Sep 18, 2017 at 8:04 AM, Herbert Valerio Riedel <hvriedel at gmail.com>
wrote:

> Hi GHC devs,
>
> A long-standing common problem when testing/dogfooding GHC HEAD is that
> at some point during the development cycle more and more packages from
> Hackage will run into build failures.
>
> Obviously, constantly nagging upstream maintainers to release quickfixes
> for unreleased GHC HEAD snapshots which will likely break again a few
> weeks later (as things are generally in flux until freeze comes into
> effect) does not scale and only introduces a latency/coordination
> bottleneck, and on top of it ultimately results in spamming the primary
> Hackage Package index with releases (which has non-negligible negative
> impact/costs of its own on the Hackage infrastructure, and thus ought to
> be minimised).
>
> OTOH, we need the ability to easily test, debug, profile, and prototype
> changes to GHC HEAD while things are still in motion, and case in point,
> if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
> run into a dozen or so of packages not building with GHC HEAD.
>
> To this end, I've finally found time to work on a side-project (related
> to matrix.hackage.haskell.org) which implements a scheme tailored to
> `cabal new-build`, which is inspired by how Eta copes with a very
> related issue (they rely on it for stable versions of the compiler);
> i.e., they maintain a set of patches at
>
>   https://github.com/typelead/eta-hackage/tree/master/patches
>
> which fix up existing Hackage packages to work with the Eta compiler.
>
>
> And this gave me the idea to use a similar scheme for GHC HEAD:
>
>   https://github.com/hvr/head.hackage/tree/master/patches
>
> This folder already contains several of patches (which mostly originate
> from Ryan, Ben and myself) to packages which I needed to patch in order
> to build popular Hackage packages & tools.
>
> The main difference is how those patches are applied; Eta uses a
> modified `cabal` which they renamed to `etlas` which is checks
> availability of .patch & .cabal files in the GitHub repo linked above;
>
> Whereas for GHC HEAD with `cabal new-build` a different scheme makes
> more sense: we simply generate an add-on Hackage repo, and use the
> existing `cabal` facilities (e.g. multi-repo support or the nix-style
> package store which makes sure that unofficially patched packages don't
> contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
> repo containing fixed up packages.
>
>
>
> I've tried to describe how to use the HEAD.hackage add-on repo in the
> README at
>
>   https://github.com/hvr/head.hackage#how-to-use
>
>
> And finally, here's a practical example of how you can use it to build
> e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
> build your project of choice; please refer to
>
>   http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html
>
> to learn more about how to describe your project via  `cabal.project`
> files):
>
>
> 0.) This assumes you have a recent cabal 2.1 snapshot built from Git
>
> 1.) create & `cd` into a new work-folder
>
> 2.) invoke `head.hackage.sh update` to update the HEAD.hackage index cache
>
> 3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
>     configuration which locally activates the HEAD.hackage overlay repo
>
> 4.) If needed, edit the cabal.project file and change where GHC
>     HEAD can be found (the script currently assumes GHC HEAD is
>     installed from my Ubuntu PPA), e.g.
>
>       with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2
>
>     or you can add something like `optional-packages: deps/*/*.cabal`
>     to have cabal pick up package source-trees unpacked in the deps/
>     folder, or you can inject ghc-options, relax upper bounds via
>     `allow-newer: *:base` etc (please refer to the cabal user guide)
>
> 5.) Create a `dummy.cabal` file (in future we will have `cabal
>     new-install` or other facilities, but for now we use this
>     workaround):
>
>
> --8<---------------cut here---------------start------------->8---
> name:                dummy
> version:             0
> build-type:          Simple
> cabal-version:       >=2.0
>
> library
>   default-language:    Haskell2010
>
>   -- library components you want cabal to solve & build for
>   -- and become accessible via .ghc.environment files and/or
>   -- `cabal new-repl`
>   build-depends: base, lens
>
>   -- executable components you want cabal to build
>   build-tool-depends: pandoc:pandoc
>
> --8<---------------cut here---------------end--------------->8---
>
>  6.) invoke `cabal new-build`
>
>  7.) If everything works, you'll find the `pandoc:pandoc` executable
>      somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
>      (you can use http://hackage.haskell.org/package/cabal-plan
>       to conveniently list the location via `cabal-plan list-bins`)
>
>  8.) As for libraries, you can either use `cabal new-repl`
>      or you can leverage GHC's package environment files:
>
>      `cabal new-build` will have generated a file like
>
>        .ghc.environment.x86_64-linux-8.3.20170913
>
>      which brings into scope all transitive dependencies of
>      `build-depends: base, lens`
>
>      Now all you need to do is simply call
>
>        ghc-stage2 --make MyTestProg.hs
>
>      to compile a program against those libs, or start up GHCi via
>
>        ghc-stage2 --interactive
>
>      and you'll be thrown into that package environment.
>
>
>
> I hope you find this useful
>
> Cheers,
>   Herbert
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>

-- 
Rahul Muttineni
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20170918/3726803e/attachment-0001.html>