Introducing bugs that are caught by slow validate?

[Evan Laforge]

Where I work we use marge, and I wound up adding a patch to run an
especially expensive "on merge only" CI run.  The two pieces are:

- patch marge to add a --require-ci-run-by-me flag, which makes marge
want to see a CI completion by margebot and start one otherwise

- modify the .gitlab-ci.yml to put the expensive CI under 'only:
variables: [$GITLAB_USER_LOGIN == "marge-bot"]'

It has the side-effect that marge-bot insists on running an extra CI
on each merge (since she no longer wants to reuse your CI run), but
since we have caching, we only pay for extra tests implied by the
heavy version of the CI run.

You could probably get the same thing by making the test outputs shake
targets, at which point it should only run tests affected by your
changes, and reuse test outputs if there hasn't been a change.  I'm
not sure if ghc tests do that... but last I recall it was a monolithic
"test everything" script that always ran all the tests.

On Thu, Apr 4, 2019 at 1:21 AM Ömer Sinan Ağacan <omeragacan at gmail.com> wrote:
>
> Hi all,
>
> I just run slow validate and it actually catches bugs (#16529) and some
> configuration problems in the test definitions (#!700). These are not caught by
> the CI jobs because they run normal validate instead of slow.
>
> (Actually they don't even run the validate script, but they run "test" instead
> of "slowtest", which I think is the main difference between normal validate and
> slow validate)
>
> As far as I can see we have nightly job definitions in .gitlab-ci.yml, which do
> slow validate, but I'm not sure if those are used?
>
> Would it be possible to switch over to slow validate in CI jobs? If slow
> validate takes too long on the CI servers, could we maybe configure CI tasks so
> that if the MR submitter is Marge, we run slow validate, otherwise we run fast
> validate?
>
> The problem with nightly jobs (assuming they're used) is that it's too late by
> the time they catch a test failure because the commits are already in the tree,
> and because nothing's blocked the bugs are more likely to stay unfixed.
>
> Ömer
> _______________________________________________
> ghc-devs mailing list
> ghc-devs at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs