Testing GHC against Hackage via CI

[Ben Gamari]

Hello everyone,

As many of you know, recently I have been working on bringing up
infrastructure for using CI-produced binary distributions to build a
subset of Hackage using the head.hackage patchset.

Happily, this effort has now converged on a usable result, embodied in
two merge requests:

 * https://gitlab.haskell.org/ghc/head.hackage/merge_requests/2
 * https://gitlab.haskell.org/ghc/ghc/merge_requests/465

ghc/head.hackage!2 adds CI support to head.hackage. In addition to the
usual pass/fail status, this job produces (e.g. [1]) a few additional
products:

 * a JSON summary of the run, describing the dependency graph and
   pass/fail state of each package. we can feed to an external service to
   track newly-failing packages.

 * a (rendered) graphviz graph showing the dependency structure of the
   built packages, each node colored by its pass/fail state

 * a tarball of build logs, each including statistics from
   -ddump-timings. The hope here is that we can use this to track
   compiler performance on real-world code

ghc/ghc!465 adds a job to GHC's own CI pipeline to trigger a
head.hackage job using the binary distribution produced earlier in the
pipeline. This job will run automatically under a variety of
circumstances:

 * via the scheduled nightly pipeline

 * on merge requests labelled with the `user-facing` label

 * when manually started using a button on the MR's Pipelines tab

Currently the head.hackage MR tests only a small number of Hackage
packages (and their dependencies). Specifically, we currently build
`aeson`, `criterion`, `singletons`, `servant`, and `scotty`. This pulls
in 127 packages in total.

There are a few things that remain to be done:

 * Better document the existence of the job and how it is triggered

 * Document how to update the list of tested packages

 * Work out how to handle tracking of persistent breakage (e.g. we want
   a notification when a package initially breaks but not in later
   builds)

 * Automatically push patched packages to a Hackage repository (e.g.
   head.hackage.org) as a result of CI.

Cheers,

- Ben


[1] https://gitlab.haskell.org/ghc/head.hackage/-/jobs/38022
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20190305/0b05a926/attachment.sig>