[GHC DevOps Group] Help with S3 costs

[Ben Gamari]

Ben Gamari <ben at well-typed.com> writes:

> Hi everyone,
>
> Our GitLab service currently uses Amazon S3 to back storage of
> artifacts and backups. At the moment we retain binary distributions for
> all builds for two weeks. This is proving to be a significant cost
> center, primarily due to transfer costs. For instance, our artifacts S3
> bucket is currently around 400 GB and this month the transfer costs
> alone will be nearly 100 USD. Thankfully, the cost of the byte-hours is
> comparatively small, being less than 10 USD.
>
> Currently I am paying for this personally but admittedly the cost has
> been a bit higher than I had anticipated. I see a few avenues which I
> think we should pursue:
>
>  * we can reduce the volume of builds that we preserve; I have started
>    (!187) with the obvious step of only preserving builds from the
>    `master` branch although we could probably pare down more.
>
>  * we could just move artifact storage back to block storage on
>    Packet.net. This storage is free to us and worked quite well before
>    we switched to S3. We could keep backups on S3 to ensure they are in
>    safe keeping.
>
>  * someone on the ghc-devops group would be able to pick up the S3 tab.
>
As it turns out there may be a fourth option:

 * Reduce our bandwidth requirements without sacrificing services.

It turns out, this is quite feasible: a user in #ghc noticed that our
previous bandwidth requirements were likely due to the fact that
gitlab-runner was downloading artifacts far more often than necessary
[1]. Fixing this is trivial (!188) and should nearly eliminate our
transfer expenses.

Of course, it would still be great if we could work out a way to cover
this cost in the future. I'd imagine it will be on the order of 10
USD/month unless something changes.

Cheers,

- Ben


[1] Specifically, the `dependencies` field defaults to a job depending
    on all of the jobs in the preceding stage
    (https://docs.gitlab.com/ee/ci/yaml/#dependencies).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 487 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/ghc-devops-group/attachments/20190122/97025b91/attachment.sig>