Gitlab's disk full again

[Ömer Sinan Ağacan]

The disk issue is fixed now, but one day later Gitlab is broken again. This time
I'm getting random 502s.

I tired to check logs, I don't know what the problem is but I saw one error line
which may be relevant

    Aug 11 12:21:09 gitlab.haskell.org unicorn[3605]:
/nix/store/h6ppx34ccb3binw7awbphaicv5q938z5-ruby2.5.5-prometheus-client-mmap-0.9.8/lib/ruby/gems/2.5.0/gems/prometheus-client-mmap-0.9.8/lib/prometheus/client/mmaped_dict.rb:42:
[BUG] Bus Error at 0x00007fe3cd6f6000

Ömer

Ömer Sinan Ağacan <omeragacan at gmail.com>, 10 Ağu 2019 Cmt, 11:48
tarihinde şunu yazdı:
>
> Hi,
>
> Just yesterday Gitlab was giving 500 because the disk was full. Ben deleted some
> files, but in less than 24h it's full again. This started happening regularly, I
> wonder if we could do something about this.
>
> The reason this time seems to be that Gitlab started generating 22G-large
> backups daily since the 7th. I'm not sure how important those backups are so I'm
> not deleting them.
>
> There's also a large docker-registry directory (101G).
>
> I think it might be good to set up some kind of downtime monitoring or maybe
> something on the Gitlab server to send an email when the disk is nearly full. It
> could send an email to people who has access to the server.
>
> It'd also be good to come up with an action plan when this happens. I have
> access to the server, but I have no idea which files are important. Documenting
> Gitlab setup (and the server details) in more details might be helpful.
>
> Does anyone have any other ideas to keep the server running?
>
> Ömer