ANN: Overlay Hackage Package Index for GHC HEAD

Mon Sep 18 13:00:13 UTC 2017

Sounds amazing - could you add the procedure you described to the "stock"
multi-GHC Travis example [1]?
It already has a "HEAD" env (see "env: BUILD=cabal GHCVER=head"...), but
breaks [2] due to the package problems you mentioned.

[1]:
https://raw.githubusercontent.com/commercialhaskell/stack/master/doc/travis-complex.yml
[2]: for haskell-indexer:
https://travis-ci.org/robinp/haskell-indexer/builds/276778630

2017-09-18 14:31 GMT+02:00 Matthew Pickering <matthewtpickering at gmail.com>:

> Something like this is definitely useful for testing.
>
> When building GHC HEAD I override the ghcHEAD derivation on nixpkgs to
> the right commit I want to use and then can similarly specify which
> patches and versions of packages to use by modifying the
> 'configuration-ghc-head.nix' file.
> This is quite a bit more flexible than just patch files as I can point
> to specific commits in git repos etc.
>
> This repo will definitely be useful for me with this workflow as well.
>
> Thanks,
>
> Matt
>
> On Mon, Sep 18, 2017 at 1:04 PM, Herbert Valerio Riedel
> <hvriedel at gmail.com> wrote:
> > Hi GHC devs,
> >
> > A long-standing common problem when testing/dogfooding GHC HEAD is that
> > at some point during the development cycle more and more packages from
> > Hackage will run into build failures.
> >
> > Obviously, constantly nagging upstream maintainers to release quickfixes
> > for unreleased GHC HEAD snapshots which will likely break again a few
> > weeks later (as things are generally in flux until freeze comes into
> > effect) does not scale and only introduces a latency/coordination
> > bottleneck, and on top of it ultimately results in spamming the primary
> > Hackage Package index with releases (which has non-negligible negative
> > impact/costs of its own on the Hackage infrastructure, and thus ought to
> > be minimised).
> >
> > OTOH, we need the ability to easily test, debug, profile, and prototype
> > changes to GHC HEAD while things are still in motion, and case in point,
> > if you try to e.g. build `pandoc` with GHC HEAD today, you'll currently
> > run into a dozen or so of packages not building with GHC HEAD.
> >
> > To this end, I've finally found time to work on a side-project (related
> > to matrix.hackage.haskell.org) which implements a scheme tailored to
> > `cabal new-build`, which is inspired by how Eta copes with a very
> > related issue (they rely on it for stable versions of the compiler);
> > i.e., they maintain a set of patches at
> >
> >   https://github.com/typelead/eta-hackage/tree/master/patches
> >
> > which fix up existing Hackage packages to work with the Eta compiler.
> >
> >
> > And this gave me the idea to use a similar scheme for GHC HEAD:
> >
> >   https://github.com/hvr/head.hackage/tree/master/patches
> >
> > This folder already contains several of patches (which mostly originate
> > from Ryan, Ben and myself) to packages which I needed to patch in order
> > to build popular Hackage packages & tools.
> >
> > The main difference is how those patches are applied; Eta uses a
> > modified `cabal` which they renamed to `etlas` which is checks
> > availability of .patch & .cabal files in the GitHub repo linked above;
> >
> > Whereas for GHC HEAD with `cabal new-build` a different scheme makes
> > more sense: we simply generate an add-on Hackage repo, and use the
> > existing `cabal` facilities (e.g. multi-repo support or the nix-style
> > package store which makes sure that unofficially patched packages don't
> > contaminate "normal" install-plans, etc.) to opt into the opt-in Hackage
> > repo containing fixed up packages.
> >
> >
> >
> > I've tried to describe how to use the HEAD.hackage add-on repo in the
> > README at
> >
> >   https://github.com/hvr/head.hackage#how-to-use
> >
> >
> > And finally, here's a practical example of how you can use it to build
> > e.g. the `pandoc` executable with GHC HEAD (can easily be adapted to
> > build your project of choice; please refer to
> >
> >   http://cabal.readthedocs.io/en/latest/nix-local-build-overview.html
> >
> > to learn more about how to describe your project via  `cabal.project`
> > files):
> >
> >
> > 0.) This assumes you have a recent cabal 2.1 snapshot built from Git
> >
> > 1.) create & `cd` into a new work-folder
> >
> > 2.) invoke `head.hackage.sh update` to update the HEAD.hackage index
> cache
> >
> > 3.) invoke `head.hackage.sh init` to create an initial `cabal.project`
> >     configuration which locally activates the HEAD.hackage overlay repo
> >
> > 4.) If needed, edit the cabal.project file and change where GHC
> >     HEAD can be found (the script currently assumes GHC HEAD is
> >     installed from my Ubuntu PPA), e.g.
> >
> >       with-compiler: /home/hvr/src/ghc-dev/inplace/bin/ghc-stage2
> >
> >     or you can add something like `optional-packages: deps/*/*.cabal`
> >     to have cabal pick up package source-trees unpacked in the deps/
> >     folder, or you can inject ghc-options, relax upper bounds via
> >     `allow-newer: *:base` etc (please refer to the cabal user guide)
> >
> > 5.) Create a `dummy.cabal` file (in future we will have `cabal
> >     new-install` or other facilities, but for now we use this
> >     workaround):
> >
> >
> > --8<---------------cut here---------------start------------->8---
> > name:                dummy
> > version:             0
> > build-type:          Simple
> > cabal-version:       >=2.0
> >
> > library
> >   default-language:    Haskell2010
> >
> >   -- library components you want cabal to solve & build for
> >   -- and become accessible via .ghc.environment files and/or
> >   -- `cabal new-repl`
> >   build-depends: base, lens
> >
> >   -- executable components you want cabal to build
> >   build-tool-depends: pandoc:pandoc
> >
> > --8<---------------cut here---------------end--------------->8---
> >
> >  6.) invoke `cabal new-build`
> >
> >  7.) If everything works, you'll find the `pandoc:pandoc` executable
> >      somewhere in your ~/.cabal/store/ghc-8.3.*/ folder
> >      (you can use http://hackage.haskell.org/package/cabal-plan
> >       to conveniently list the location via `cabal-plan list-bins`)
> >
> >  8.) As for libraries, you can either use `cabal new-repl`
> >      or you can leverage GHC's package environment files:
> >
> >      `cabal new-build` will have generated a file like
> >
> >        .ghc.environment.x86_64-linux-8.3.20170913
> >
> >      which brings into scope all transitive dependencies of
> >      `build-depends: base, lens`
> >
> >      Now all you need to do is simply call
> >
> >        ghc-stage2 --make MyTestProg.hs
> >
> >      to compile a program against those libs, or start up GHCi via
> >
> >        ghc-stage2 --interactive
> >
> >      and you'll be thrown into that package environment.
> >
> >
> >
> > I hope you find this useful
> >
> > Cheers,
> >   Herbert
> > _______________________________________________
> > ghc-devs mailing list
> > ghc-devs at haskell.org
> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20170918/1ec06d6f/attachment-0001.html>