[GHC DevOps Group] Stackage HEAD, now working

[Karpov, Mark]

Hello,

Last year there was an initiative to execute Stackage nightly build plans
with development versions of GHC and by comparing results that we obtain
with different commits we could detect how they affect packages in our
ecosystem. Thus we potentially can detect regressions and other suspicious
changes.

* The original blog post introducing the initiative:
https://www.tweag.io/posts/2017-10-27-stackage-head.html
* A talk from HIW'17: https://www.youtube.com/watch?v=UAI-mplXUkY

Recently we resumed working on this and actually I'm happy to announce that
Stackage HEAD is functional now. In fact, a bit unexpectedly, it detected
something today:

https://ghc.haskell.org/trac/ghc/ticket/15022

This is not necessarily a bug or regression, but still fewer test suites of
stm-delay build at 3cfb12d8adac37e5565d66fd173e4648cc041e65 than before
at 5161609117c16cb7b29b2b8b1cd41e74341d4137 which is a suspicious change to
Stackage HEAD build report manager.

There is a short description how it works here:
https://github.com/tweag/stackage-head#how-it-works

The logic that determines if a change in status of a package is suspicious
is here:

https://github.com/tweag/stackage-head/blob/master/Stackage/HEAD/BuildDiff.hs#L61-L94

And the entire source code of the tool is available here:

https://github.com/tweag/stackage-head

As you can see, a package can either fail to build, or be unreachable (due
to failing dependencies) or it can succeed in which case we record number
of succeeding/failing test suites.

Currently statistics look like this (with nightly-2018-04-05):

* Failing packages: 5
* Unreachable packages: 794
* Packages that build: 897

The build takes about one hour.

Circle CI logs are available here:

https://circleci.com/gh/tweag/stackage-head

I configured this to run 4 times per day for a start. I'll receive email
notifications on failures which I plan to convert to trac tickets manually
(for now.).

I could not find a way to make Circle CI send emails to other people like
Ben, but I see there are various hook available, so we could use some of
them if desirable.

Best,

Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devops-group/attachments/20180411/0de9b425/attachment.html>