[GHC DevOps Group] CircleCI status and budget concerns

[Simon Peyton Jones]

|  I agree that (a) is useful short-term fix, but I think, the only reasonable
|  long-term option is (c). After all, one reason to go with a hosted CI
|  solution was to be able to solve this kind of problem with financial support
|  by one of the stakeholders.

I agree with Manuel.  $50/month is *miniscule* when compared to the cost of
person-hours required to work around the problem.  If we can solve problems
with very small amounts of money, that's a gift -- we should grab it!  We have
lots of problems that can't be so simply solved 😊.

Simon

|  -----Original Message-----
|  From: Ghc-devops-group <ghc-devops-group-bounces at haskell.org> On Behalf Of
|  Manuel M T Chakravarty
|  Sent: 18 June 2018 05:04
|  To: Ben Gamari <ben at well-typed.com>
|  Cc: GHC DevOps Group <ghc-devops-group at haskell.org>
|  Subject: Re: [GHC DevOps Group] CircleCI status and budget concerns
|  
|  Hi Ben,
|  
|  Thanks for bringing this up.
|  
|  I agree that (a) is useful short-term fix, but I think, the only reasonable
|  long-term option is (c). After all, one reason to go with a hosted CI
|  solution was to be able to solve this kind of problem with financial support
|  by one of the stakeholders.
|  
|  How many containers have we got? (I think, one needs GitHub org admin access
|  to be able to see that.)
|  
|  Cheers,
|  Manuel
|  
|  > Am 16.06.2018 um 03:13 schrieb Ben Gamari <ben at well-typed.com>:
|  >
|  > Signierter PGP-Teil
|  > Hi everyone,
|  >
|  > Over the last few weeks while finalizing the 8.6 branch I've been
|  > working on cleaning up some loose ends on the CI front. This includes,
|  >
|  > * Fixing the Fedora and Hadrian builds
|  > * Whittling away at the i386 Linux failures
|  > * Incorporating nightly slow validation (which is now green, thanks to
|  >   Alp's efforts)
|  > * Incorporating Alp's fixes to Hadrian to get this build green again
|  >
|  > During this process, however, I have noticed that a concerning banner
|  > has appeared at the top of the CircleCI interface. It currently reads:
|  >
|  >   Your current usage represents 84% of your ghc Linux plan's limit.
|  >   Please upgrade in order to ensure no disruption in building.
|  >
|  > With the percentage gradually ticking upwards as builds finish. I
|  > believe 84% refers to our plan's container time allotment of 1500
|  > hours (confusingly mislabeled as "1500 minutes" on the website). So
|  > far in the current May 21 - Jun 21 billing period we have used 1250
|  > hours of build time. This is concerning as it presumably means that
|  > when we eventually overrun our budget we will be left without any CI
|  > at all for the remainder of the month.
|  >
|  > Given that we are currently building only a subset of commits on only
|  > the `master` I'm rather concerned that we are within a stone's throw
|  > of this situation (perhaps even going over this month). This seems
|  > untenable, especially when we consider that the number of builds will
|  > soon be growing to include stable branch backports, patches under
|  > review, and release builds.
|  >
|  > To manage this it seems that we will likely need to draw from a few of
|  > the tools at our disposal:
|  >
|  > a. Reduce the number of build configurations that we build every commit
|  >    in (e.g. Fedora and LLVM could likely be relegated to a nightly
|  > build)
|  >
|  > b. Reduce the amount of parallelism that we allow CircleCI to employ:
|  >    having access to CI consistently throughout the month is far
|  >    preferable to faster build turnarounds for some of the month
|  >    followed by periods of no CI for the rest.
|  >
|  > c. Increase the build time limit; this is $50/month/container
|  >
|  > In the short-run we can implement (a) immediately. This of course will
|  > reduce our coverage, but it looks like our coverage goals may not be
|  > practical given the resources we have. I've looked into (b) and
|  > haven't found any indication that CircleCI supports our use-case. If
|  > this is the case then we will have to simply pursue (c).
|  >
|  > I suppose haskell.org may be able to contribute here. However, I
|  > wonder what other resources could be deployed.
|  >
|  > Cheers,
|  >
|  > - Ben
|  >
|  >
|  
|  _______________________________________________
|  Ghc-devops-group mailing list
|  Ghc-devops-group at haskell.org
|  https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group