[Haskell-cafe] [ANN] Stratosphere: AWS CloudFormation EDSL in Haskell

[David Reaver]

Thanks Alberto!

You can certainly set up permissions and security groups on EC2 nodes using
CloudFormation, run your job, and then tear down the CloudFormation stack.
You can use stratosphere to specify the AWS resources like an EC2 instance,
a VPC for it, some security groups, and an instance role. Then, you can use
something like Ansible or even just a bash script to set up and run your
job on that machine. Once it's done, you can tear down the stack you just
built.

We actually do something similar to what you said for running pgbadger on
our Postgres RDS logs. Our logs can reach a few GB per hour, so downloading
them to a local machine can be pretty time-consuming, and processing can
take long as well. In order to get RDS logs, you need to have permissions
to do so on AWS. I made a stack in stratosphere that creates a large EC2
instance for processing, a strict security group that locks down the
instance, and also sets up a temporary user that only has permission to
download RDS logs. Using Ansible, I then install pgbadger and its
dependencies, download the logs, run pgbadger, and then download the
results. Once the job is done, everything that was just created is
destroyed.

In summary, what you're saying is indeed possible and is a great use-case
for CloudFormation. You just need to using something on top of
stratosphere, as this library is simply used to build the CloudFormation
templates. I'm considering adding support for running the templates
directly in this library, maybe using amazonka-cloudformation. So far that
hasn't been necessary for us.

Also note that CloudFormation has a way to estimate the cost of your
created resources:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-paying.html

On Tue, Apr 19, 2016 at 12:21 PM, Alberto G. Corona <agocorona at gmail.com>
wrote:

> Very nice job.
>
> I have tried AWS EC2 and I failed miserably since the permission system
> and his intricate configurations, his paranoid security, and  their secret
> formulas for pricing are beyond an average human intelligence.
>
> It is possible to have an example stratosphere application that could
> configure and run a Haskell program in one or many EC2 nodes , close the
> instances and send back the results?
>
> Are there  some elements still lacking to achieve such goal?
>
> 2016-04-19 20:56 GMT+02:00 David Reaver <johndreaver at gmail.com>:
>
>> Github: https://github.com/frontrowed/stratosphere
>> Hackage: https://hackage.haskell.org/package/stratosphere
>>
>> CloudFormation is a system that provisions and updates Amazon Web Services
>> (AWS) resources based on declarative templates. Common criticisms of
>> CloudFormation include the use of JSON as the template language and
>> limited
>> error-checking, often only available in the form of run-time errors and
>> stack
>> rollbacks. By wrapping templates in Haskell, we are able to easily
>> construct
>> them and help ensure correctness.
>>
>> The goals of stratosphere are to:
>> - Build a Haskell EDSL to specify CloudFormation templates. Since it is
>>   embedded in Haskell, it is type-checked and generally much easier to
>> work
>>   with than raw JSON.
>> - Have a simple checking/linting system outside of the types that can find
>>   common errors in templates.
>> - Be able to also read valid CloudFormation JSON templates so they can be
>>   type-checked. This also gives us free integration tests by using the
>> huge
>>   amount of example templates available in the AWS docs.
>>
>> Most of the commonly used CloudFormation resources are implemented, and
>> adding
>> new resources is very straightforward. (We created a web scraper that
>> generates
>> a JSON file from a given CloudFormation documentation page URL, and from
>> that
>> we generate a Haskell module.) So far, we have implemented resources we
>> use at
>> Front Row Education, and we will add more resources over time.
>>
>> The library is very much in a usable state as-is. However, we want to
>> make any
>> sweeping changes while it is still young. If you have any suggestions at
>> all,
>> you want us to add your favorite resource, or if you find a bug, please
>> file an
>> issue on Github!
>>
>> Also, we want to give a huge thanks to Brendan Hay, the author of
>> amazonka, for
>> his ideas and feedback on the project.
>>
>> _______________________________________________
>> Haskell-Cafe mailing list
>> Haskell-Cafe at haskell.org
>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe
>>
>>
>
>
> --
> Alberto.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20160419/d3ab9a70/attachment.html>