<div dir="ltr"><div>Thanks Manuel for putting together the requirements.</div><div><br></div><div>Is there a way to make the choice more data-driven? I realize many things are very hard to estimate in advance, order of magnitude swags could still be informative. Off the cuff it would be useful to have the numbers of builds and their latencies, numbers of binary package downloads, prices of different options, expected human operations time, how long the solution is required to work before it is to be reconsidered.<br></div><div><br></div><div>Given such numbers one could pose an optimization problem and compare the options.</div><div><br></div><div>Thanks</div><div>Greg</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Oct 12, 2017 at 6:18 AM Ben Gamari <<a href="mailto:ben@well-typed.com">ben@well-typed.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Manuel M T Chakravarty <<a href="mailto:manuel.chakravarty@tweag.io" target="_blank">manuel.chakravarty@tweag.io</a>> writes:<br>
<br>
> As promised, I have taken a first cut at listing the requirements and<br>
> the pros and cons of the main contenders on a Trac page:<br>
><br>
> <a href="https://ghc.haskell.org/trac/ghc/wiki/ContinuousIntegration" rel="noreferrer" target="_blank">https://ghc.haskell.org/trac/ghc/wiki/ContinuousIntegration</a><br>
><br>
I think this list is being a bit generous to the hosted option.<br>
<br>
Other costs of this approach might include:<br>
<br>
* Under this heterogeneous scheme we will have to maintain two or more<br>
distinct CI systems, each requiring some degree of setup and<br>
maintenance.<br>
<br>
* Using qemu for building on/for a non-Linux/amd64 platforms requires a<br>
non-negligible amount of additional complexity (see rust's CI<br>
implementation [1])<br>
<br>
* It's unclear whether testing GHC via qemu is even practical given<br>
computational constraints.<br>
<br>
* We lose the ability to prioritize jobs, requiring more hardware to<br>
maintain similar build turnaround<br>
<br>
* We are utterly dependent on our CI service(s) to behave well; for<br>
instance, here are two examples that the Rust infrastructure team<br>
related to me,<br>
<br>
* They have been struggling to keep Travis the tail of their build<br>
turnaround time distribution in check, with some builds taking<br>
over 8 hours to complete. Despite raising the issue with Travis<br>
customer support they are still having trouble, despite being a<br>
paying customer.<br>
<br>
* They have noticed that Travis has a tendency to simply drop builds<br>
in mid-flight, losing hours of work. Again, despite working with<br>
upstream they haven't been able to resolve the problem<br>
<br>
* They have been strongly affected by apparent instability in<br>
Travis' OS X infrastructure which goes down, to quote, "*a lot*"<br>
<br>
Of course, both of these are picking on Travis in particular as that<br>
is the example we have available. However, in general the message<br>
here is that by giving up our own infrastructure we are at the mercy<br>
of the services that we use. Unfortunately, sometimes those services<br>
are not accustomed to testing projects of the scale of GHC or rustc.<br>
At this point you have little recourse but to minimize the damage.<br>
<br>
We avoid all of this by self-hosting (at, of course, the expense of<br>
administration time). Furthermore, we continue to benefit from hardware<br>
provided by a multitude of sources including users, Rackspace (and other<br>
VPS providers if we wanted), and programs like OSU OSL. It is important<br>
to remember that until recently we were operating under the assumption<br>
that these were the only resources available to us for testing.<br>
<br>
It's still quite unclear to me what a CircleCI/Appveyor solution will<br>
ultimately cost, but will almost certainly not be free. Assuming there<br>
are users who are willing to foot that bill, this is of course fine.<br>
However, it's quite contrary to the assumptions we have been working<br>
with for much of this process.<br>
<br>
<br>
Lastly: If I understand the point correctly, the "the set up is not<br>
forkable" "con" of Jenkins is not accurate. Under Jenkins the build<br>
configuration resides in the repository being tested. A user can easily<br>
modify it and submit a PR, which will be tested just like any other<br>
change.<br>
<br>
<br>
[1] <a href="https://github.com/rust-lang/rust/tree/master/src/ci" rel="noreferrer" target="_blank">https://github.com/rust-lang/rust/tree/master/src/ci</a><br>
<br>
<br>
> Maybe I am biased, but is there any advantage to Jenkins other than<br>
> that we can run builds and tests on exotic platforms?<br>
<br>
Some of these "exotic" platforms might also be called "the most populous<br>
architecture in the world" (ARM), "the operating system that feeds a<br>
third of the world's Internet traffic (FreeBSD), and "the operating<br>
system that powers much of the world's financial system" (AIX). I'm not<br>
sure that the "exotic" label really does these platforms justice.<br>
<br>
More importantly, all of these platforms have contributors working on<br>
their support in GHC. Historically, GHC HQ has tried to recognize their<br>
efforts by allowing porters to submit binary distributions which are<br>
distributed alongside GHC HQ distributions. Recently I have tried to<br>
pursue a different model, handling some of these binary builds myself in<br>
the name of consistency and reduced release overhead (as previously we<br>
incurred a full round-trip through binary build contributors every time<br>
we released).<br>
<br>
The desire to scale our release process up to handle the breadth of<br>
platforms that GHC supports, with either Tier 1 or what is currently<br>
Tier 2 support, was one motivation for the new CI effort. While I don't<br>
consider testing any one of these platforms to be a primary goal, I do<br>
think it is important to have a viable plan by which they might be<br>
covered in the future for this reason.<br>
<br>
<br>
To be clear, I am supportive of the CI-as-a-service direction. However,<br>
I want to recognize the trade-offs where they exist and have answers to<br>
some of the thorny questions, including those surrounding platform<br>
support, before committing.<br>
<br>
Cheers,<br>
<br>
- Ben<br>
_______________________________________________<br>
Ghc-devops-group mailing list<br>
<a href="mailto:Ghc-devops-group@haskell.org" target="_blank">Ghc-devops-group@haskell.org</a><br>
<a href="https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group" rel="noreferrer" target="_blank">https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group</a><br>
</blockquote></div>