[Haskell-cafe] [ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

[Brendan Hay]

Great work, David. Congratulations on the release!

On 19 April 2016 at 21:39, David Reaver <johndreaver at gmail.com> wrote:

> Thanks Alberto!
>
> You can certainly set up permissions and security groups on EC2 nodes
> using CloudFormation, run your job, and then tear down the CloudFormation
> stack. You can use stratosphere to specify the AWS resources like an EC2
> instance, a VPC for it, some security groups, and an instance role. Then,
> you can use something like Ansible or even just a bash script to set up and
> run your job on that machine. Once it's done, you can tear down the stack
> you just built.
>
> We actually do something similar to what you said for running pgbadger on
> our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading
> them to a local machine can be pretty time-consuming, and processing can
> take long as well. In order to get RDS logs, you need to have permissions
> to do so on AWS. I made a stack in stratosphere that creates a large EC2
> instance for processing, a strict security group that locks down the
> instance, and also sets up a temporary user that only has permission to
> download RDS logs. Using Ansible, I then install pgbadger and its
> dependencies, download the logs, run pgbadger, and then download the
> results. Once the job is done, everything that was just created is
> destroyed.
>
> In summary, what you're saying is indeed possible and is a great use-case
> for CloudFormation. You just need to using something on top of
> stratosphere, as this library is simply used to build the CloudFormation
> templates. I'm considering adding support for running the templates
> directly in this library, maybe using amazonka-cloudformation. So far that
> hasn't been necessary for us.
>
> Also note that CloudFormation has a way to estimate the cost of your
> created resources:
> http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html
>
> On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona <agocorona at gmail.com>
> wrote:
>
>> Very nice job.
>>
>> I have tried AWS EC2 and I failed miserably since the permission system
>> and his intricate configurations, his paranoid security, and  their secret
>> formulas for pricing are beyond an average human intelligence.
>>
>> It is possible to have an example stratosphere application that could
>> configure and run a Haskell program in one or many EC2 nodes , close the
>> instances and send back the results?
>>
>> Are there  some elements still lacking to achieve such goal?
>>
>> 2016-04-19 20:56 GMT+02:00 David Reaver <johndreaver at gmail.com>:
>>
>>> Github: https://github.com/frontrowed/stratosphere
>>> Hackage: https://hackage.haskell.org/package/stratosphere
>>>
>>> CloudFormation is a system that provisions and updates Amazon Web
>>> Services
>>> (AWS) resources based on declarative templates. Common criticisms of
>>> CloudFormation include the use of JSON as the template language and
>>> limited
>>> error-checking, often only available in the form of run-time errors and
>>> stack
>>> rollbacks. By wrapping templates in Haskell, we are able to easily
>>> construct
>>> them and help ensure correctness.
>>>
>>> The goals of stratosphere are to:
>>> - Build a Haskell EDSL to specify CloudFormation templates. Since it is
>>>   embedded in Haskell, it is type-checked and generally much easier to
>>> work
>>>   with than raw JSON.
>>> - Have a simple checking/linting system outside of the types that can
>>> find
>>>   common errors in templates.
>>> - Be able to also read valid CloudFormation JSON templates so they can be
>>>   type-checked. This also gives us free integration tests by using the
>>> huge
>>>   amount of example templates available in the AWS docs.
>>>
>>> Most of the commonly used CloudFormation resources are implemented, and
>>> adding
>>> new resources is very straightforward. (We created a web scraper that
>>> generates
>>> a JSON file from a given CloudFormation documentation page URL, and from
>>> that
>>> we generate a Haskell module.) So far, we have implemented resources we
>>> use at
>>> Front Row Education, and we will add more resources over time.
>>>
>>> The library is very much in a usable state as-is. However, we want to
>>> make any
>>> sweeping changes while it is still young. If you have any suggestions at
>>> all,
>>> you want us to add your favorite resource, or if you find a bug, please
>>> file an
>>> issue on Github!
>>>
>>> Also, we want to give a huge thanks to Brendan Hay, the author of
>>> amazonka, for
>>> his ideas and feedback on the project.
>>>
>>> _______________________________________________
>>> Haskell-Cafe mailing list
>>> Haskell-Cafe at haskell.org
>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>>>
>>>
>>
>>
>> --
>> Alberto.
>>
>
>
> _______________________________________________
> Haskell-Cafe mailing list
> Haskell-Cafe at haskell.org
> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160420/146b1dba/attachment.html>