Scaling back CI (for now)?

[Matthew Pickering]

Hi all,

Everyone has probably noticed that getting anything merged is a real
effort at the moment. The main problem is that CI takes in the region
of 5-7 hours and then spuriously fails at the end. After 5-7 hours you
have to rebase and run CI again and so on. Therefore I propose to run
just these four jobs on every MR:

validate-x86_64-linux-deb9
validate-x86_64-linux-deb8-hadrian
validate-x86_64-windows
validate-x86_64-darwin

The reasoning is as follows:

validate-x86_64-linux-deb9
validate-x86_64-linux-deb8-hadrian

These run first and are reliable and finish within an hour. Then we
have lots of less reliable, lower priority jobs.

Two windows jobs which take forever to run.

validate-x86_64-windows
validate-x86_64-windows-hadrian

One darwin job

validate-x86_64-darwin

Many more linux jobs

validate-x86_64-linux-deb9-unreg
validate-x86_64-linux-deb9-integer-simple
validate-x86_64-linux-fedora27
validate-x86_64-linux-deb9-llvm
validate-x86_64-linux-deb8
validate-i386-linux-deb9
validate-aarch64-linux-deb9

So I don't argue that these are important to test but at the moment
they produce too much friction on every commit through a combination
of lack of resources and taking too long.

Further to this, we really don't need to test fedora27, deb9 and deb8
for every build. When was the last time we broke one of these
platforms but not the other, it's rare!

So the concrete proposal is to slim back the per commit validation to four jobs.

validate-x86_64-linux-deb9
validate-x86_64-linux-deb8-hadrian
validate-x86_64-windows
validate-x86_64-darwin

which will test on the three major platforms.

All the other flavours should be run once the commit reaches master.

Thoughts?

Cheers,

Matt