[GHC DevOps Group] CircleCI status and budget concerns

[Ben Gamari]

Hi everyone,

Over the last few weeks while finalizing the 8.6 branch I've been
working on cleaning up some loose ends on the CI front. This includes,

 * Fixing the Fedora and Hadrian builds
 * Whittling away at the i386 Linux failures
 * Incorporating nightly slow validation (which is now green, thanks to
   Alp's efforts)
 * Incorporating Alp's fixes to Hadrian to get this build green again

During this process, however, I have noticed that a concerning banner
has appeared at the top of the CircleCI interface. It currently reads:

   Your current usage represents 84% of your ghc Linux plan's limit.
   Please upgrade in order to ensure no disruption in building.

With the percentage gradually ticking upwards as builds finish. I
believe 84% refers to our plan's container time allotment of 1500
hours (confusingly mislabeled as "1500 minutes" on the website). So far
in the current May 21 - Jun 21 billing period we have used 1250 hours of
build time. This is concerning as it presumably means that when we
eventually overrun our budget we will be left without any CI at all for
the remainder of the month.

Given that we are currently building only a subset of commits on only
the `master` I'm rather concerned that we are within a stone's throw of
this situation (perhaps even going over this month). This seems
untenable, especially when we consider that the number of builds will
soon be growing to include stable branch backports, patches under
review, and release builds.

To manage this it seems that we will likely need to draw from a few of
the tools at our disposal:

 a. Reduce the number of build configurations that we build every commit
    in (e.g. Fedora and LLVM could likely be relegated to a nightly build)

 b. Reduce the amount of parallelism that we allow CircleCI to employ:
    having access to CI consistently throughout the month is far
    preferable to faster build turnarounds for some of the month
    followed by periods of no CI for the rest.

 c. Increase the build time limit; this is $50/month/container

In the short-run we can implement (a) immediately. This of course will
reduce our coverage, but it looks like our coverage goals may not be
practical given the resources we have. I've looked into (b) and haven't
found any indication that CircleCI supports our use-case. If this is
the case then we will have to simply pursue (c).

I suppose haskell.org may be able to contribute here. However, I wonder
what other resources could be deployed.

Cheers,

- Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devops-group/attachments/20180615/f70a0e58/attachment.sig>