[GHC DevOps Group] CI effort status

Mon Feb 5 03:35:36 UTC 2018

Hi Gershom,

Ben is surely the main actor and he has put considerable effort into this. However, we (Tweag) did help out initially writing some of the original CI configurations.

Having said that, it would be absolutely fabulous if other developers could help out. Please let Ben and me know if you know anybody who would be happy to help!

Cheers,
Manuel

> 05.02.2018 13:41 Gershom B <gershomb at gmail.com>:
> 
> A question from an observer here -- my understanding was that part of
> the plan with the shift in  CI infrastructure was that the burden
> would be lifted from Ben's exclusive shoulders here and there would be
> some greater division of labor, which is made possible in part by
> using shared standard services rather than self-hosted  solutions. But
> at the moment I see reports largely of Ben continuing to try to
> resolve issues and move this plan forward on his own. Is there still
> some medium-term plan have a more collective effort in this
> transition?
> 
> --Gershom
> 
> 
> On Sat, Feb 3, 2018 at 12:28 PM, Ben Gamari <ben at well-typed.com> wrote:
>> Ben Gamari <ben at well-typed.com> writes:
>> 
>>> Manuel M T Chakravarty <manuel.chakravarty at tweag.io> writes:
>>> 
>>>> Hi Ben,
>>>> 
>>>> I meant to post on https://github.com/appveyor/ci/issues/517
>>>> <https://github.com/appveyor/ci/issues/517> to request an increased
>>>> limit, but didn’t get around to it yet. If you’d be able to put a
>>>> request on that issue, that’d be great.
>>>> 
>>> Sure, I'm on it.
>>> 
>> I have created a new GHCAppveyor (due to name length constraints)
>> Appveyor [1] project and configured it to pull from the ghc/ghc GitHub
>> mirror. I have also requested, and was granted, the typical build time
>> limit extension to 90 minutes.
>> 
>> Unfortunately, it seems that even 90 minutes is insufficient to even
>> finish a build, much less run the testsuite, under Appveyor's build
>> environment. Given where the build was terminated, I would guess that
>> it would need at least another 10 minutes of compilation to make it to
>> the testsuite. On top of this the testsuite will require another ~35
>> minutes (as it is quite heavy on process spawning, which is very
>> expensive on Windows).
>> 
>> I haven't yet inquired as to whether a further build time extension
>> would be possible. However, I am not hopeful that our plan of using
>> Appveyor will be feasible without purchasing build time.
>> 
>> 
>> On the CircleCI front, I have been continuing work to clear up the
>> remaining build failures. At this point only two remain:
>> 
>> * I have a patch (D4360) to fix T11489 by running our build jobs as an
>>  unprivileged user
>> 
>> * scc01 appears to be slightly non-deterministic; I am investigating
>>  this.
>> 
>> Unfortunately the CircleCI infrastructure is still exhibiting a fair
>> amount of flakiness. See, for instance, this build which is shown to be
>> "Cancelled" despite having finished (and having apparently been run at
>> least twice). Judging from the build history, this seems to be a fairly
>> regular occurrence. I have contacted CircleCI about this but have not
>> yet heard back.
>> 
>> I am also occassionally seeing rather extreme variance in test times.
>> In particular the linux-llvm target usually completes in around 4 hours
>> 20 minutes, but sometimes takes over 5 hours, resulting in the build
>> timing out. It appears that the build hangs during the testsuite run
>> (e.g. [2]); it's not impossible that this is due to a bug in the
>> testsuite driver but I have been able to reproduce this neither locally
>> nor remotely on CircleCI infrastructure so it has proved to be a tough
>> nut to crack.
>> 
>> Cheers,
>> 
>> - Ben
>> 
>> [1] https://ci.appveyor.com/project/GHCAppveyor/ghc
>> [2] https://circleci.com/gh/ghc/ghc/1558
>> 
>> _______________________________________________
>> Ghc-devops-group mailing list
>> Ghc-devops-group at haskell.org
>> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group
>> 
> _______________________________________________
> Ghc-devops-group mailing list
> Ghc-devops-group at haskell.org
> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group