[GHC DevOps Group] Continuous integration effort status

[Facundo Domínguez]

Hello,

Thanks very much for the update.

> c. Artifact preservation. I recommend that we push these to S3 ...

How many artifacts should we keep? I guess all the artifacts produced
by every build of the master branch since the beginning of time is
unnecessary, right?

If going through S3, how do we protect the S3 credentials from rogue
pull requests that attempt to expose them?

Cheers,
Facundo



On Mon, Mar 5, 2018 at 2:15 PM, Ben Gamari <ben at well-typed.com> wrote:
> Hi everyone,
>
> Simon suggested that I write with a bit of a status update summarizing
> where we stand on the CI reboot effort.
>
> As of today we have the following:
>
>  * Per-push CircleCI builds [1] in a variety of ways:
>
>      * "Vanilla" amd64/Linux
>      * amd64/Linux with -fllvm
>      * Hadrian amd64/Linux
>      * "Vanilla" amd64/Darwin
>
>    These currently fail due to #14705, although this should be resolved
>    shortly.
>
>  * A 64-bit Windows build of BuildFlavour=quick on Appveyor [2]. This
>    currently fails due to a maximum artifact size restriction. Moreover,
>    a full GHC build is not practical on Appveyor's infrastructure due to
>    build time limits.
>
> To help achieve full Appveyor support, Google X has kindly agreed to
> sponsor computational resources which we can use to host our own
> instance of Appveyor's self-hosted server. In addition to the cost of
> the computational resources which Google has covered, this solution will
> require a $50/month Premium Appveyor subscription. As far as I know we
> still don't have a source for this, although it is possible that
> haskell.org could foot this bill.
>
> There are a variety of items currently in flight,
>
>  * Alp will be looking at fixing the --slow validate failures (#14890)
>
>  * David is looking at incorporating last summer's performance testsuite
>    rework, #12758, into our CircleCI infrastructure.
>
>  * For the next day or two I am rather tied up with other things, but
>    after this I will have some time to look at the boxes that Greg has
>    brought up and see what it will take to bring up Appveyor. If someone
>    else would like to pick this up then do let me know.
>
>  * Mark Karpov has done some great work pushing forward some
>    patches that I started but didn't have time to finish up. This
>    includes,
>      * Moving the CircleCI infrastructure to Docker, eliminating a
>        spurious testsuite failure [3]
>
>      * Adding a 32-bit Linux target [4]
>
> The following things are rather lacking direction at the moment,
>
>  a. Integration between Phabricator and Appveyor/CircleCI (#14506).
>
>  b. Validation of each commit in a push as both Appveyor and CircleCI
>     only build the push head (#14505). This is quite important as having
>     a full commit history has been invaluable in identifying
>     regressions. I will start pushing commits one-at-a-time until this
>     is sorted out.
>
>  c. Artifact preservation. I recommend that we push these to S3 or
>     similar to avoid relying on Appveyor/CircleCI's (generally rather
>     restrictive) artifact archival facilities.
>
>  d. Hadrian builds seem to non-deterministically hang on CircleCI
>     (#14600). I've never observed this locally and have had no luck
>     reproducing via SSH on a CircleCI box, so it's unclear what to do
>     about this.
>
>  e. Support for building on non-Debian-based systems (e.g. Fedora),
>     which is necessary if we want to produce our binary distributions
>     via CI
>
> I suspect that after (b) is fixed we will find that the computational
> resources of our CircleCI account may be a bit lacking, but we can cross
> that bridge once we get there.
>
> Cheers,
>
> - Ben
>
>
> [1] https://circleci.com/gh/ghc/ghc/tree/master
> [2] https://ci.appveyor.com/project/GHCAppveyor/ghc
> [3] https://github.com/ghc/ghc/pull/104
> [4] https://github.com/ghc/ghc/pull/106
>
> _______________________________________________
> Ghc-devops-group mailing list
> Ghc-devops-group at haskell.org
> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group
>