[GHC DevOps Group] Help with S3 costs

[Simon Peyton Jones]

I think it would probably help to summarise all the costs that we are currently incurring, and who is currently paying for them (including in-kind contributions).  That would help to give perspective.

Not incurring costs unnecessarily is obviously the first step, so your second message (with good news about reducing storage costs) is great.  If the remaining monthly cost still looks high, we should review whether we want to keep everything we are keeping.  Does anyone ever look at this stuff?

You should not have to pay for anything personally!

Simon

| -----Original Message-----
| From: Ghc-devops-group <ghc-devops-group-bounces at haskell.org> On Behalf Of
| Ben Gamari
| Sent: 22 January 2019 18:41
| To: ghc-devops-group at haskell.org
| Subject: [GHC DevOps Group] Help with S3 costs
| 
| Hi everyone,
| 
| Our GitLab service currently uses Amazon S3 to back storage of
| artifacts and backups. At the moment we retain binary distributions for
| all builds for two weeks. This is proving to be a significant cost
| center, primarily due to transfer costs. For instance, our artifacts S3
| bucket is currently around 400 GB and this month the transfer costs
| alone will be nearly 100 USD. Thankfully, the cost of the byte-hours is
| comparatively small, being less than 10 USD.
| 
| Currently I am paying for this personally but admittedly the cost has
| been a bit higher than I had anticipated. I see a few avenues which I
| think we should pursue:
| 
|  * we can reduce the volume of builds that we preserve; I have started
|    (!187) with the obvious step of only preserving builds from the
|    `master` branch although we could probably pare down more.
| 
|  * we could just move artifact storage back to block storage on
|    Packet.net. This storage is free to us and worked quite well before
|    we switched to S3. We could keep backups on S3 to ensure they are in
|    safe keeping.
| 
|  * someone on the ghc-devops group would be able to pick up the S3 tab.
| 
| Thoughts?
| 
| Cheers,
| 
| - Ben