[GHC DevOps Group] CircleCI job accounting question

[Manuel M T Chakravarty]

[Continuing to pick up the threads left dormant while I was travelling.]

Hi Ben,

As per #14506, we still need to find a solution to triggering CircleCI builds from Harbormaster, right? Just to make sure that I understand the requirements correctly, from what you wrote, Harbormaster needs a separate (from the main GHC repo) Git repo where contributors push patches (I guess, via the arc tool) for review and that is what you called the staging area. Moreover, CircleCI requires the staging area repo to be on GitHub. Is that correct?

Concerning the push permission (which would need to be manually enabled on GitHub), are you saying that the repo would have to be push for all? Wouldn’t it be possible to use a GitHub API key or so?

I found this feature request, which supposedly has been complete. Does that help?

  https://discuss.circleci.com/t/compatibility-with-phabricator/183

I also found this

  https://github.com/signalfx/phabricator-circleci

which has the disadvantage that it involves using AWS Lambda.

Cheers,
Manuel


> Am 23.11.2017 um 02:31 schrieb Ben Gamari <ben at well-typed.com>:
> 
> Manuel M T Chakravarty <manuel.chakravarty at tweag.io> writes:
> 
>>> Ben Gamari <ben at well-typed.com>:
>>> 
>>> Manuel M T Chakravarty <manuel.chakravarty at tweag.io> writes:
>>> 
>>>> Mateusz had a first stab
>>>> 
>>>> https://github.com/tweag/ghc/blob/tweag/circleci-macos/appveyor.yml
>>>> 
>>>> but got stuck in the default resource limits. We emailed them with a
>>>> request, but there was no answer so far. I’ll follow up on it.
>>>> 
>>> Any update on this? For the record, I have confirmed with the Rustaceans
>>> that Mozilla indeed pays for their usage.
>> 
>> No, sorry, I have been completely taken out with travelling and
>> conference for the last week. (Just arrived in the Netherlands.)
> 
> Quite alright; just wondering.
> 
>>> * It appears that CircleCI only builds the head commits of pushes.
>>>  Making this configurable has been a feature request for nearly a year
>>>  now, so it looks like we will need to work around this. I briefly
>>>  looked into setting up some automation to trigger builds on otherwise
>>>  untested commits, but ran into apparent API bugginess. It looks like
>>>  we'll just need to ensure that contributors push at most one commit
>>>  at a time for now to ensure all commits get testing. See GHC #14505
>>>  for details.
>> 
>> Why do we need the intermediate builds exactly? Wouldn’t they usually
>> fail? (When I do PRs with multiple commits, the state of the tree
>> between this commits will usually not be well-defined.)
> 
> No, every commit should build. This is in part a difference between
> Phabricator's patch-based model and GitHub's feature branch model.
> However, many projects using the latter also demand that all
> intermediate commits must be atomic, buildable changes. Sacrificing this
> property greatly complicates bisection.
> 
> Building all intermediates is desireable as ultimately we would like to
> preserve per-commit build artifacts for the last few months of commits
> to enable easy bisection.
> 
>>> * I have tried enabling testing of Harbormaster Differentials via
>>>  CircleCI. Unfortunately it appears that CircleCI only supports
>>>  testing repositories hosted on GitHub. There are a few ways in which
>>>  we could proceed,
>>> 
>>>   a. Move ghc's staging area (the repository where Arcanist pushes
>>>      patches submitted with `arc diff`) to GitHub. This, however, would
>>>      require that we manually manage push privileges to this repository.
>> 
>> What do you mean by manually manage push privileges? In what way is
>> that not manually at the moment?
> 
> As long as a user had added a key to their Phabricator account pushing
> to the staging area will "just work". It requires no intervention from
> me. I believe in the event that we moved to GitHub I would need to
> manually grant users commit privileges to the staging area.
> 
>>>   b. Try to work around the issue by mirroring GHC's staging area to
>>>      GitHub and manually trigger CircleCI builds.
>> 
>> Is the manual triggering necessary, because Harbormaster would need to
>> wait until the repo is triggered (which it can’t)?
> 
> In general this whole mirroring situation doesn't appear to be a
> use-case that Phabricator's CircleCI integration supports. It demands
> that the staging area of the tested repository is hosted on GitHub.
> 
>>> * I have been honing the Hadrian test infrastructure; I'm currently
>>>  waiting on a build, but I expect this attempt will pass, at which point
>>>  I will merge it.
>> 
>> Great!
> 
> Sadly the build appears to reliably hang with no output. I'll need to
> look into this.
> 
> Cheers,
> 
> - Ben