Gitlab's disk full again
Ömer Sinan Ağacan
omeragacan at gmail.com
Sun Aug 11 16:22:45 UTC 2019
The disk issue is fixed now, but one day later Gitlab is broken again. This time
I'm getting random 502s.
I tired to check logs, I don't know what the problem is but I saw one error line
which may be relevant
Aug 11 12:21:09 gitlab.haskell.org unicorn:
[BUG] Bus Error at 0x00007fe3cd6f6000
Ömer Sinan Ağacan <omeragacan at gmail.com>, 10 Ağu 2019 Cmt, 11:48
tarihinde şunu yazdı:
> Just yesterday Gitlab was giving 500 because the disk was full. Ben deleted some
> files, but in less than 24h it's full again. This started happening regularly, I
> wonder if we could do something about this.
> The reason this time seems to be that Gitlab started generating 22G-large
> backups daily since the 7th. I'm not sure how important those backups are so I'm
> not deleting them.
> There's also a large docker-registry directory (101G).
> I think it might be good to set up some kind of downtime monitoring or maybe
> something on the Gitlab server to send an email when the disk is nearly full. It
> could send an email to people who has access to the server.
> It'd also be good to come up with an action plan when this happens. I have
> access to the server, but I have no idea which files are important. Documenting
> Gitlab setup (and the server details) in more details might be helpful.
> Does anyone have any other ideas to keep the server running?
More information about the ghc-devs