From ben at well-typed.com Mon Mar 5 17:15:44 2018 From: ben at well-typed.com (Ben Gamari) Date: Mon, 05 Mar 2018 12:15:44 -0500 Subject: [GHC DevOps Group] Continuous integration effort status Message-ID: <87r2oy4f5w.fsf@smart-cactus.org> Hi everyone, Simon suggested that I write with a bit of a status update summarizing where we stand on the CI reboot effort. As of today we have the following: * Per-push CircleCI builds [1] in a variety of ways: * "Vanilla" amd64/Linux * amd64/Linux with -fllvm * Hadrian amd64/Linux * "Vanilla" amd64/Darwin These currently fail due to #14705, although this should be resolved shortly. * A 64-bit Windows build of BuildFlavour=quick on Appveyor [2]. This currently fails due to a maximum artifact size restriction. Moreover, a full GHC build is not practical on Appveyor's infrastructure due to build time limits. To help achieve full Appveyor support, Google X has kindly agreed to sponsor computational resources which we can use to host our own instance of Appveyor's self-hosted server. In addition to the cost of the computational resources which Google has covered, this solution will require a $50/month Premium Appveyor subscription. As far as I know we still don't have a source for this, although it is possible that haskell.org could foot this bill. There are a variety of items currently in flight, * Alp will be looking at fixing the --slow validate failures (#14890) * David is looking at incorporating last summer's performance testsuite rework, #12758, into our CircleCI infrastructure. * For the next day or two I am rather tied up with other things, but after this I will have some time to look at the boxes that Greg has brought up and see what it will take to bring up Appveyor. If someone else would like to pick this up then do let me know. * Mark Karpov has done some great work pushing forward some patches that I started but didn't have time to finish up. This includes, * Moving the CircleCI infrastructure to Docker, eliminating a spurious testsuite failure [3] * Adding a 32-bit Linux target [4] The following things are rather lacking direction at the moment, a. Integration between Phabricator and Appveyor/CircleCI (#14506). b. Validation of each commit in a push as both Appveyor and CircleCI only build the push head (#14505). This is quite important as having a full commit history has been invaluable in identifying regressions. I will start pushing commits one-at-a-time until this is sorted out. c. Artifact preservation. I recommend that we push these to S3 or similar to avoid relying on Appveyor/CircleCI's (generally rather restrictive) artifact archival facilities. d. Hadrian builds seem to non-deterministically hang on CircleCI (#14600). I've never observed this locally and have had no luck reproducing via SSH on a CircleCI box, so it's unclear what to do about this. e. Support for building on non-Debian-based systems (e.g. Fedora), which is necessary if we want to produce our binary distributions via CI I suspect that after (b) is fixed we will find that the computational resources of our CircleCI account may be a bit lacking, but we can cross that bridge once we get there. Cheers, - Ben [1] https://circleci.com/gh/ghc/ghc/tree/master [2] https://ci.appveyor.com/project/GHCAppveyor/ghc [3] https://github.com/ghc/ghc/pull/104 [4] https://github.com/ghc/ghc/pull/106 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From facundo.dominguez at tweag.io Tue Mar 6 14:18:38 2018 From: facundo.dominguez at tweag.io (=?UTF-8?Q?Facundo_Dom=C3=ADnguez?=) Date: Tue, 6 Mar 2018 11:18:38 -0300 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: <87r2oy4f5w.fsf@smart-cactus.org> References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: Hello, Thanks very much for the update. > c. Artifact preservation. I recommend that we push these to S3 ... How many artifacts should we keep? I guess all the artifacts produced by every build of the master branch since the beginning of time is unnecessary, right? If going through S3, how do we protect the S3 credentials from rogue pull requests that attempt to expose them? Cheers, Facundo On Mon, Mar 5, 2018 at 2:15 PM, Ben Gamari wrote: > Hi everyone, > > Simon suggested that I write with a bit of a status update summarizing > where we stand on the CI reboot effort. > > As of today we have the following: > > * Per-push CircleCI builds [1] in a variety of ways: > > * "Vanilla" amd64/Linux > * amd64/Linux with -fllvm > * Hadrian amd64/Linux > * "Vanilla" amd64/Darwin > > These currently fail due to #14705, although this should be resolved > shortly. > > * A 64-bit Windows build of BuildFlavour=quick on Appveyor [2]. This > currently fails due to a maximum artifact size restriction. Moreover, > a full GHC build is not practical on Appveyor's infrastructure due to > build time limits. > > To help achieve full Appveyor support, Google X has kindly agreed to > sponsor computational resources which we can use to host our own > instance of Appveyor's self-hosted server. In addition to the cost of > the computational resources which Google has covered, this solution will > require a $50/month Premium Appveyor subscription. As far as I know we > still don't have a source for this, although it is possible that > haskell.org could foot this bill. > > There are a variety of items currently in flight, > > * Alp will be looking at fixing the --slow validate failures (#14890) > > * David is looking at incorporating last summer's performance testsuite > rework, #12758, into our CircleCI infrastructure. > > * For the next day or two I am rather tied up with other things, but > after this I will have some time to look at the boxes that Greg has > brought up and see what it will take to bring up Appveyor. If someone > else would like to pick this up then do let me know. > > * Mark Karpov has done some great work pushing forward some > patches that I started but didn't have time to finish up. This > includes, > * Moving the CircleCI infrastructure to Docker, eliminating a > spurious testsuite failure [3] > > * Adding a 32-bit Linux target [4] > > The following things are rather lacking direction at the moment, > > a. Integration between Phabricator and Appveyor/CircleCI (#14506). > > b. Validation of each commit in a push as both Appveyor and CircleCI > only build the push head (#14505). This is quite important as having > a full commit history has been invaluable in identifying > regressions. I will start pushing commits one-at-a-time until this > is sorted out. > > c. Artifact preservation. I recommend that we push these to S3 or > similar to avoid relying on Appveyor/CircleCI's (generally rather > restrictive) artifact archival facilities. > > d. Hadrian builds seem to non-deterministically hang on CircleCI > (#14600). I've never observed this locally and have had no luck > reproducing via SSH on a CircleCI box, so it's unclear what to do > about this. > > e. Support for building on non-Debian-based systems (e.g. Fedora), > which is necessary if we want to produce our binary distributions > via CI > > I suspect that after (b) is fixed we will find that the computational > resources of our CircleCI account may be a bit lacking, but we can cross > that bridge once we get there. > > Cheers, > > - Ben > > > [1] https://circleci.com/gh/ghc/ghc/tree/master > [2] https://ci.appveyor.com/project/GHCAppveyor/ghc > [3] https://github.com/ghc/ghc/pull/104 > [4] https://github.com/ghc/ghc/pull/106 > > _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group at haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group > From marlowsd at gmail.com Tue Mar 6 15:19:38 2018 From: marlowsd at gmail.com (Simon Marlow) Date: Tue, 6 Mar 2018 15:19:38 +0000 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: <87r2oy4f5w.fsf@smart-cactus.org> References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: Thanks for the great summary, Ben. On 5 March 2018 at 17:15, Ben Gamari wrote: > > The following things are rather lacking direction at the moment, > > a. Integration between Phabricator and Appveyor/CircleCI (#14506). > What are our options here? I presume we'll have to write some custom integration? > b. Validation of each commit in a push as both Appveyor and CircleCI > only build the push head (#14505). This is quite important as having > a full commit history has been invaluable in identifying > regressions. I will start pushing commits one-at-a-time until this > is sorted out. > > c. Artifact preservation. I recommend that we push these to S3 or > similar to avoid relying on Appveyor/CircleCI's (generally rather > restrictive) artifact archival facilities. > > d. Hadrian builds seem to non-deterministically hang on CircleCI > (#14600). I've never observed this locally and have had no luck > reproducing via SSH on a CircleCI box, so it's unclear what to do > about this. > > e. Support for building on non-Debian-based systems (e.g. Fedora), > which is necessary if we want to produce our binary distributions > via CI > > I suspect that after (b) is fixed we will find that the computational > resources of our CircleCI account may be a bit lacking, but we can cross > that bridge once we get there. > I think the same will be true of (a). Cheers Simon > Cheers, > > - Ben > > > [1] https://circleci.com/gh/ghc/ghc/tree/master > [2] https://ci.appveyor.com/project/GHCAppveyor/ghc > [3] https://github.com/ghc/ghc/pull/104 > [4] https://github.com/ghc/ghc/pull/106 > > _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group at haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m at tweag.io Tue Mar 6 15:37:45 2018 From: m at tweag.io (Boespflug, Mathieu) Date: Tue, 6 Mar 2018 16:37:45 +0100 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: Hi Ben, good summary of the current directions the Group is pursuing. > * For the next day or two I am rather tied up with other things, but > after this I will have some time to look at the boxes that Greg has > brought up and see what it will take to bring up Appveyor. If someone > else would like to pick this up then do let me know. Mark Karpov has started work on this today, using Tweag I/O's Premium subscription for Appveyor until a subscription can be sorted out for the ghc org. Best, -- Mathieu Boespflug Founder at http://tweag.io. On 6 March 2018 at 16:19, Simon Marlow wrote: > Thanks for the great summary, Ben. > > On 5 March 2018 at 17:15, Ben Gamari wrote: >> >> >> The following things are rather lacking direction at the moment, >> >> a. Integration between Phabricator and Appveyor/CircleCI (#14506). > > > What are our options here? I presume we'll have to write some custom > integration? > >> >> b. Validation of each commit in a push as both Appveyor and CircleCI >> only build the push head (#14505). This is quite important as having >> a full commit history has been invaluable in identifying >> regressions. I will start pushing commits one-at-a-time until this >> is sorted out. >> >> c. Artifact preservation. I recommend that we push these to S3 or >> similar to avoid relying on Appveyor/CircleCI's (generally rather >> restrictive) artifact archival facilities. >> >> d. Hadrian builds seem to non-deterministically hang on CircleCI >> (#14600). I've never observed this locally and have had no luck >> reproducing via SSH on a CircleCI box, so it's unclear what to do >> about this. >> >> e. Support for building on non-Debian-based systems (e.g. Fedora), >> which is necessary if we want to produce our binary distributions >> via CI >> >> I suspect that after (b) is fixed we will find that the computational >> resources of our CircleCI account may be a bit lacking, but we can cross >> that bridge once we get there. > > > I think the same will be true of (a). > > Cheers > Simon > > >> >> Cheers, >> >> - Ben >> >> >> [1] https://circleci.com/gh/ghc/ghc/tree/master >> [2] https://ci.appveyor.com/project/GHCAppveyor/ghc >> [3] https://github.com/ghc/ghc/pull/104 >> [4] https://github.com/ghc/ghc/pull/106 >> >> _______________________________________________ >> Ghc-devops-group mailing list >> Ghc-devops-group at haskell.org >> https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group >> > > > _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group at haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group > From ben at well-typed.com Tue Mar 6 16:51:05 2018 From: ben at well-typed.com (Ben Gamari) Date: Tue, 06 Mar 2018 11:51:05 -0500 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: <87bmg1407f.fsf@smart-cactus.org> Facundo Domínguez writes: > Hello, > > Thanks very much for the update. > >> c. Artifact preservation. I recommend that we push these to S3 ... > > How many artifacts should we keep? I guess all the artifacts produced > by every build of the master branch since the beginning of time is > unnecessary, right? > Right. There are a few reasons we want to keep artifacts: a. To allow contributors to, e.g., build and view their documentation changes via CI b. To allow us to retrieve build artifacts from CI for releases c. As reference builds for bisection In the case of (a) and (b) I think it would be sufficient to keep a week or two of builds. (c) is a bit trickier. Ideally we would have an archive with a sort of decaying retention policy; e.g., keep all builds from the last month, every fourth build for the last 3 months, every eighth for the last 6 months, etc. > If going through S3, how do we protect the S3 credentials from rogue > pull requests that attempt to expose them? > Indeed that is a great question. I'll admit I don't have a great answer for this at the moment. It's certainly not an easy problem. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From manuel.chakravarty at tweag.io Wed Mar 7 00:41:42 2018 From: manuel.chakravarty at tweag.io (Manuel M T Chakravarty) Date: Wed, 7 Mar 2018 11:41:42 +1100 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: <87r2oy4f5w.fsf@smart-cactus.org> References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: Thanks, Ben. That is a very helpful summary. Manuel > 06.03.2018 04:15 Ben Gamari : > > Hi everyone, > > Simon suggested that I write with a bit of a status update summarizing > where we stand on the CI reboot effort. > > As of today we have the following: > > * Per-push CircleCI builds [1] in a variety of ways: > > * "Vanilla" amd64/Linux > * amd64/Linux with -fllvm > * Hadrian amd64/Linux > * "Vanilla" amd64/Darwin > > These currently fail due to #14705, although this should be resolved > shortly. > > * A 64-bit Windows build of BuildFlavour=quick on Appveyor [2]. This > currently fails due to a maximum artifact size restriction. Moreover, > a full GHC build is not practical on Appveyor's infrastructure due to > build time limits. > > To help achieve full Appveyor support, Google X has kindly agreed to > sponsor computational resources which we can use to host our own > instance of Appveyor's self-hosted server. In addition to the cost of > the computational resources which Google has covered, this solution will > require a $50/month Premium Appveyor subscription. As far as I know we > still don't have a source for this, although it is possible that > haskell.org could foot this bill. > > There are a variety of items currently in flight, > > * Alp will be looking at fixing the --slow validate failures (#14890) > > * David is looking at incorporating last summer's performance testsuite > rework, #12758, into our CircleCI infrastructure. > > * For the next day or two I am rather tied up with other things, but > after this I will have some time to look at the boxes that Greg has > brought up and see what it will take to bring up Appveyor. If someone > else would like to pick this up then do let me know. > > * Mark Karpov has done some great work pushing forward some > patches that I started but didn't have time to finish up. This > includes, > * Moving the CircleCI infrastructure to Docker, eliminating a > spurious testsuite failure [3] > > * Adding a 32-bit Linux target [4] > > The following things are rather lacking direction at the moment, > > a. Integration between Phabricator and Appveyor/CircleCI (#14506). > > b. Validation of each commit in a push as both Appveyor and CircleCI > only build the push head (#14505). This is quite important as having > a full commit history has been invaluable in identifying > regressions. I will start pushing commits one-at-a-time until this > is sorted out. > > c. Artifact preservation. I recommend that we push these to S3 or > similar to avoid relying on Appveyor/CircleCI's (generally rather > restrictive) artifact archival facilities. > > d. Hadrian builds seem to non-deterministically hang on CircleCI > (#14600). I've never observed this locally and have had no luck > reproducing via SSH on a CircleCI box, so it's unclear what to do > about this. > > e. Support for building on non-Debian-based systems (e.g. Fedora), > which is necessary if we want to produce our binary distributions > via CI > > I suspect that after (b) is fixed we will find that the computational > resources of our CircleCI account may be a bit lacking, but we can cross > that bridge once we get there. > > Cheers, > > - Ben > > > [1] https://circleci.com/gh/ghc/ghc/tree/master > [2] https://ci.appveyor.com/project/GHCAppveyor/ghc > [3] https://github.com/ghc/ghc/pull/104 > [4] https://github.com/ghc/ghc/pull/106 > _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group at haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group From gershomb at gmail.com Mon Mar 12 17:18:28 2018 From: gershomb at gmail.com (Gershom B) Date: Mon, 12 Mar 2018 13:18:28 -0400 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: <87r2oy4f5w.fsf@smart-cactus.org> References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: On Mon, Mar 5, 2018 at 12:15 PM, Ben Gamari wrote: > To help achieve full Appveyor support, Google X has kindly agreed to > sponsor computational resources which we can use to host our own > instance of Appveyor's self-hosted server. In addition to the cost of > the computational resources which Google has covered, this solution will > require a $50/month Premium Appveyor subscription. As far as I know we > still don't have a source for this, although it is possible that > haskell.org could foot this bill. Since nobody else seemed forthcoming here, I ran it by the haskell.org committee and we think it would be a reasonable use of our funds to cover this. Whenever you actually need to set up payment, just contact Ryan and he'll be able to provide information for billing to the committee directly.. -g From simonpj at microsoft.com Mon Mar 12 21:31:12 2018 From: simonpj at microsoft.com (Simon Peyton Jones) Date: Mon, 12 Mar 2018 21:31:12 +0000 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: Thank you Gershom! Simon -----Original Message----- From: Ghc-devops-group [mailto:ghc-devops-group-bounces at haskell.org] On Behalf Of Gershom B Sent: 12 March 2018 17:18 To: Ben Gamari Cc: ghc-devops-group at haskell.org Subject: Re: [GHC DevOps Group] Continuous integration effort status On Mon, Mar 5, 2018 at 12:15 PM, Ben Gamari wrote: > To help achieve full Appveyor support, Google X has kindly agreed to > sponsor computational resources which we can use to host our own > instance of Appveyor's self-hosted server. In addition to the cost of > the computational resources which Google has covered, this solution > will require a $50/month Premium Appveyor subscription. As far as I > know we still don't have a source for this, although it is possible > that haskell.org could foot this bill. Since nobody else seemed forthcoming here, I ran it by the haskell.org committee and we think it would be a reasonable use of our funds to cover this. Whenever you actually need to set up payment, just contact Ryan and he'll be able to provide information for billing to the committee directly.. -g _______________________________________________ Ghc-devops-group mailing list Ghc-devops-group at haskell.org https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group From ben at well-typed.com Mon Mar 12 22:37:53 2018 From: ben at well-typed.com (Ben Gamari) Date: Mon, 12 Mar 2018 18:37:53 -0400 Subject: [GHC DevOps Group] Continuous integration effort status In-Reply-To: References: <87r2oy4f5w.fsf@smart-cactus.org> Message-ID: <876060zzra.fsf@smart-cactus.org> Gershom B writes: > On Mon, Mar 5, 2018 at 12:15 PM, Ben Gamari wrote: > >> To help achieve full Appveyor support, Google X has kindly agreed to >> sponsor computational resources which we can use to host our own >> instance of Appveyor's self-hosted server. In addition to the cost of >> the computational resources which Google has covered, this solution will >> require a $50/month Premium Appveyor subscription. As far as I know we >> still don't have a source for this, although it is possible that >> haskell.org could foot this bill. > > Since nobody else seemed forthcoming here, I ran it by the haskell.org > committee and we think it would be a reasonable use of our funds to > cover this. > Thanks Gershom! However, I think we shouldn't let haskell.org's generous offer stop us from exploring additional funding options. The more the merrier. Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: From mark.karpov at tweag.io Tue Mar 20 15:14:11 2018 From: mark.karpov at tweag.io (Karpov, Mark) Date: Tue, 20 Mar 2018 22:14:11 +0700 Subject: [GHC DevOps Group] A solution for storing build artifacts Message-ID: Hello, I'd like to share a solution for storing build artifacts we've come up with. The solution is to use S3 to store N last artifacts per job on Circle CI and also on AppVeyor. Auth info can be stored in environment variables if we're careful to use the following settings: * For Circle CI, there is "Pass secrets to builds from forked pull requests", which should be set to "off". This way only pull requests from the same repo will have access to sensitive environment variables. * For AppVeyor, similarly, there is "Enable secure variables in pull requests from the same repository only" (should be enabled). Then we can put the key that AWS uses for authentication in an environment variable and either * Use AWS CLI utility and do some bash coding around it. * Create a little application that will handle uploading to S3 with all the necessary logic. By the logic I mean that we perhaps should do some clean-up, such as deletion of old files. Let me know if this sounds good to you, and if so, I could start on this. Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From gnezdo at x.team Tue Mar 20 20:38:40 2018 From: gnezdo at x.team (Greg Steuck (Sh-toy-k)) Date: Tue, 20 Mar 2018 13:38:40 -0700 Subject: [GHC DevOps Group] U2Fs for GHC devops Message-ID: I have obtained a few U2F devices from Google Advanced Protection program . I sent one to Ben and I am happy to send more to GHC devops. I used such devices for authenticating to GitHub and Google, they make a good second factor. Feel free to send me the addresses+phone numbers and I will ship with Fedex. Thanks Greg -------------- next part -------------- An HTML attachment was scrubbed... URL: From manuel.chakravarty at tweag.io Thu Mar 22 02:02:46 2018 From: manuel.chakravarty at tweag.io (Manuel M T Chakravarty) Date: Thu, 22 Mar 2018 13:02:46 +1100 Subject: [GHC DevOps Group] A solution for storing build artifacts In-Reply-To: References: Message-ID: Jonas, our resident DevOps guru is happy with this approach, so I am, too ;) Jonas notes that ”If the garbage collection can be time based, S3 can handle it itself: https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html ” Cheers, Manuel > 21.03.2018 02:14 Karpov, Mark : > > Hello, > > I'd like to share a solution for storing build artifacts we've come up with. The solution is to use S3 to store N last artifacts per job on Circle CI and also on AppVeyor. Auth info can be stored in environment variables if we're careful to use the following settings: > > * For Circle CI, there is "Pass secrets to builds from forked pull requests", which should be set to "off". This way only pull requests from the same repo will have access to sensitive environment variables. > * For AppVeyor, similarly, there is "Enable secure variables in pull requests from the same repository only" (should be enabled). > > Then we can put the key that AWS uses for authentication in an environment variable and either > > * Use AWS CLI utility and do some bash coding around it. > * Create a little application that will handle uploading to S3 with all the necessary logic. > > By the logic I mean that we perhaps should do some clean-up, such as deletion of old files. > > Let me know if this sounds good to you, and if so, I could start on this. > > Cheers, > > Mark > _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group at haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group -------------- next part -------------- An HTML attachment was scrubbed... URL: From jonas.chevalier at tweag.io Thu Mar 22 12:20:29 2018 From: jonas.chevalier at tweag.io (Jonas Chevalier) Date: Thu, 22 Mar 2018 12:20:29 +0000 Subject: [GHC DevOps Group] A solution for storing build artifacts In-Reply-To: References: Message-ID: What we could do is have different lifecycles per folder. The /dev folder would keep artifacts for a few months (S3 is quite cheap). The /release folder which contains tagged releases would not be garbage-collected at all. On Thu, Mar 22, 2018 at 2:02 AM Manuel M T Chakravarty < manuel.chakravarty at tweag.io> wrote: > Jonas, our resident DevOps guru is happy with this approach, so I am, too > ;) > > Jonas notes that ”If the garbage collection can be time based, S3 can > handle it itself: > https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html > ” > > Cheers, > Manuel > > 21.03.2018 02:14 Karpov, Mark : > > Hello, > > I'd like to share a solution for storing build artifacts we've come up > with. The solution is to use S3 to store N last artifacts per job on Circle > CI and also on AppVeyor. Auth info can be stored in environment variables > if we're careful to use the following settings: > > * For Circle CI, there is "Pass secrets to builds from forked pull > requests", which should be set to "off". This way only pull requests from > the same repo will have access to sensitive environment variables. > * For AppVeyor, similarly, there is "Enable secure variables in pull > requests from the same repository only" (should be enabled). > > Then we can put the key that AWS uses for authentication in an environment > variable and either > > * Use AWS CLI utility and do some bash coding around it. > * Create a little application that will handle uploading to S3 with all > the necessary logic. > > By the logic I mean that we perhaps should do some clean-up, such as > deletion of old files. > > Let me know if this sounds good to you, and if so, I could start on this. > > Cheers, > > Mark > _______________________________________________ > Ghc-devops-group mailing list > Ghc-devops-group at haskell.org > https://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at well-typed.com Mon Mar 26 06:34:13 2018 From: ben at well-typed.com (Ben Gamari) Date: Mon, 26 Mar 2018 02:34:13 -0400 Subject: [GHC DevOps Group] A solution for storing build artifacts In-Reply-To: References: Message-ID: <87efk7jqfz.fsf@smart-cactus.org> "Karpov, Mark" writes: > Hello, > > I'd like to share a solution for storing build artifacts we've come up > with. The solution is to use S3 to store N last artifacts per job on Circle > CI and also on AppVeyor. Auth info can be stored in environment variables > if we're careful to use the following settings: > > * For Circle CI, there is "Pass secrets to builds from forked pull > requests", which should be set to "off". This way only pull requests from > the same repo will have access to sensitive environment variables. > * For AppVeyor, similarly, there is "Enable secure variables in pull > requests from the same repository only" (should be enabled). > > Then we can put the key that AWS uses for authentication in an environment > variable and either > > * Use AWS CLI utility and do some bash coding around it. > * Create a little application that will handle uploading to S3 with all the > necessary logic. > > By the logic I mean that we perhaps should do some clean-up, such as > deletion of old files. > > Let me know if this sounds good to you, and if so, I could start on this. > I think this sounds like a good start. It would be nice to have the ability to keep more sparse samples of older artifacts, but this is something we can address later (and perhaps via other means). Cheers, - Ben -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: not available URL: