[GHC DevOps Group] CircleCI status and budget concerns

[Manuel M T Chakravarty]

Hi Ben,

Thanks for bringing this up.

I agree that (a) is useful short-term fix, but I think, the only reasonable long-term option is (c). After all, one reason to go with a hosted CI solution was to be able to solve this kind of problem with financial support by one of the stakeholders.

How many containers have we got? (I think, one needs GitHub org admin access to be able to see that.)

Cheers,
Manuel

> Am 16.06.2018 um 03:13 schrieb Ben Gamari <ben at well-typed.com>:
> 
> Signierter PGP-Teil
> Hi everyone,
> 
> Over the last few weeks while finalizing the 8.6 branch I've been
> working on cleaning up some loose ends on the CI front. This includes,
> 
> * Fixing the Fedora and Hadrian builds
> * Whittling away at the i386 Linux failures
> * Incorporating nightly slow validation (which is now green, thanks to
>   Alp's efforts)
> * Incorporating Alp's fixes to Hadrian to get this build green again
> 
> During this process, however, I have noticed that a concerning banner
> has appeared at the top of the CircleCI interface. It currently reads:
> 
>   Your current usage represents 84% of your ghc Linux plan's limit.
>   Please upgrade in order to ensure no disruption in building.
> 
> With the percentage gradually ticking upwards as builds finish. I
> believe 84% refers to our plan's container time allotment of 1500
> hours (confusingly mislabeled as "1500 minutes" on the website). So far
> in the current May 21 - Jun 21 billing period we have used 1250 hours of
> build time. This is concerning as it presumably means that when we
> eventually overrun our budget we will be left without any CI at all for
> the remainder of the month.
> 
> Given that we are currently building only a subset of commits on only
> the `master` I'm rather concerned that we are within a stone's throw of
> this situation (perhaps even going over this month). This seems
> untenable, especially when we consider that the number of builds will
> soon be growing to include stable branch backports, patches under
> review, and release builds.
> 
> To manage this it seems that we will likely need to draw from a few of
> the tools at our disposal:
> 
> a. Reduce the number of build configurations that we build every commit
>    in (e.g. Fedora and LLVM could likely be relegated to a nightly build)
> 
> b. Reduce the amount of parallelism that we allow CircleCI to employ:
>    having access to CI consistently throughout the month is far
>    preferable to faster build turnarounds for some of the month
>    followed by periods of no CI for the rest.
> 
> c. Increase the build time limit; this is $50/month/container
> 
> In the short-run we can implement (a) immediately. This of course will
> reduce our coverage, but it looks like our coverage goals may not be
> practical given the resources we have. I've looked into (b) and haven't
> found any indication that CircleCI supports our use-case. If this is
> the case then we will have to simply pursue (c).
> 
> I suppose haskell.org may be able to contribute here. However, I wonder
> what other resources could be deployed.
> 
> Cheers,
> 
> - Ben
> 
>