Consistent CI failure in job nightly-i386-linux-deb9-validate

[Bryan Richter]

Matthew pointed out that the build system already parallelizes jobs, so
it's risky to force parallelization of any individual job. That means I
should just revert.

On Wed, Sep 28, 2022 at 2:38 PM Cheng Shao <cheng.shao at tweag.io> wrote:

> I believe we can either modify ci.sh to disable parallel compression
> for i386, or modify .gitlab/gen_ci.hs and .gitlab/jobs.yaml to disable
> XZ_OPT=-9 for i386.
>
> On Wed, Sep 28, 2022 at 1:21 PM Bryan Richter <bryan at haskell.foundation>
> wrote:
> >
> > Aha: while i386-linux-deb9-validate sets no extra XZ options,
> nightly-i386-linux-deb9-validate (the failing job) sets "XZ_OPT = 9".
> >
> > A revert would fix the problem, but presumably so would tweaking that
> option. Does anyone have information that would lead to a better decision
> here?
> >
> >
> > On Wed, Sep 28, 2022 at 2:02 PM Cheng Shao <cheng.shao at tweag.io> wrote:
> >>
> >> Sure, in which case pls revert it. Apologies for the impact, though
> >> I'm still a bit curious, the i386 job did pass in the original MR.
> >>
> >> On Wed, Sep 28, 2022 at 1:00 PM Bryan Richter <bryan at haskell.foundation>
> wrote:
> >> >
> >> > Yep, it seems to mostly be xz that is running out of memory. (All
> recent builds that I sampled, but not all builds through all time.) Thanks
> for pointing it out!
> >> >
> >> > I can revert the change.
> >> >
> >> > On Wed, Sep 28, 2022 at 11:46 AM Cheng Shao <cheng.shao at tweag.io>
> wrote:
> >> >>
> >> >> Hi Bryan,
> >> >>
> >> >> This may be an unintended fallout of !8940. Would you try starting an
> >> >> i386 pipeline with it reversed to see if it solves the issue, in
> which
> >> >> case we should revert or fix it in master?
> >> >>
> >> >> On Wed, Sep 28, 2022 at 9:58 AM Bryan Richter via ghc-devs
> >> >> <ghc-devs at haskell.org> wrote:
> >> >> >
> >> >> > Hi all,
> >> >> >
> >> >> > For the past week or so, nightly-i386-linux-deb9-validate has been
> failing consistently.
> >> >> >
> >> >> > They show up on the failure dashboard because the logs contain the
> phrase "Cannot allocate memory".
> >> >> >
> >> >> > I haven't looked yet to see if they always fail in the same place,
> but I'll do that soon. The first example I looked at, however, has the line
> "xz: (stdin): Cannot allocate memory", so it's not GHC (alone) causing the
> problem.
> >> >> >
> >> >> > As a consequence of showing up on the dashboard, the jobs get
> restarted. Since they fail consistently, they keep getting restarted. Since
> the jobs keep getting restarted, the pipelines stay alive. When I checked
> just now, there were 8 nightly runs still running. :) Thus I'm going to
> cancel the still-running nightly-i386-linux-deb9-validate jobs and let the
> pipelines die in peace. You can still find all examples of failed jobs on
> the dashboard:
> >> >> >
> >> >> >
> https://grafana.gitlab.haskell.org/d/167r9v6nk/ci-spurious-failures?orgId=2&from=now-90d&to=now&refresh=5m&var-types=cannot_allocate
> >> >> >
> >> >> > To prevent future problems, it would be good if someone could help
> me look into this. Otherwise I'll just disable the job. :(
> >> >> > _______________________________________________
> >> >> > ghc-devs mailing list
> >> >> > ghc-devs at haskell.org
> >> >> > http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/ghc-devs/attachments/20220929/e6af9403/attachment.html>