[web-devel] [Yesod] Better web app deployment

[Michael Snoyman]

On Fri, May 4, 2012 at 6:17 PM, Lyndon Maydwell <maydwell at gmail.com> wrote:
> This is absolutely on par with what I've been thinking about over the
> last week. I take it you are drawing some inspiration from the Twelve
> Factor App Manifesto that you linked me to? It certainly seems that
> way :-)
>
> I started building a management application to act as an interface for
> this kind of architecture, however, I'm stuck grappling with the Yesod
> form model as I haven't had much time to spend on it yet.
>
> A couple of ideas:
>
> Are you planning to have strong isolation between the applications
> using some kind of sand-boxing? Given an Nginx reverse proxy this
> should be made quite trivial by having the apps communicating over a
> port or socket. Just a question of how it would be implemented.

I wasn't planning on it, but you're right that it should be feasible.
I suppose I've been taking the approach so far of letting the apps
have much more freedom than say a Heroku deployment. The idea would be
that only trusted apps would be running on the system.

> I feel that a framework like this would benefit enormously from having
> strong partnership with a continuous integration framework for
> actually building the applications. This could run as an application
> inside the application deployment framework in order to be nicely
> circular! A benefit of this would be that you could guarantee that the
> build and deploy architecture is the same - freeing developers from
> the worry that their local machine is a different CPU architecture or
> operating system.

Like Greg said, the ideal would be building this on a separate virtual
machine to avoid bogging down your app server with compilation. One
possible idea would be to provide a nice, powerful EC2 instance as a
community, and get a simple setup with Github commit hooks to be able
to deploy from it automatically. I definitely want to explore this,
but after the deployment system itself is in place.

>
> Love the idea. You're always a step ahead.
>
> On Fri, May 4, 2012 at 10:55 PM, Michael Snoyman <michael at snoyman.com> wrote:
>> Hi all,
>>
>> (Note: I originally was going to just send this to the Yesod mailing list,
>> but frankly, most of the code I'm setting up would work for any web
>> framework. If people using other frameworks want to use this deployment
>> code, and are interested in working with me to make sure it's universally
>> applicable, please be in touch.)
>>
>> I'm planning on improving the deployment code starting next week. Instead of
>> comparing to what we currently have (and which I think only I use), let me
>> just give a high-level approach. Please give input on how you'd like to
>> change things. Once I have the basic structure put together, I'll put it up
>> on Github, and contributions will be greatly appreciated:
>>
>> * We'll use Nginx as a reverse proxy. This will allow us to have virtual
>> hosting, more efficient static file serving (possibly from a separate
>> domain), and zero-downtime deployments.
>> * The scaffolding will accept a number of environment variables for
>> configuring it, such as PostgreSQL connection information, approot, static
>> root, and port number. (Most of this is already implemented.)
>> * Each site will have a deployment configuration file, for things like
>> domain name, whether a PostgreSQL database is required, and any system
>> packages are required to be installed (we'll assume a Debian-based system
>> for now). The deployer will automatically create databases and provide that
>> information via environment variables.
>> * Apps will have a specific file format, which is essentially a deployment
>> config file and a bunch of things (static folder, executable, etc) it needs.
>> An app will then be a tarball of those folders.
>> * The deployer will watch a folder for changes. Deployments will be
>> performed by copying files into this folder.
>> * When a new app is deployed, it will be unpacked, started up, and
>> monitored. If it stops, it will be restarted. The Nginx config file will be
>> updated to reverse proxy to the app's port (randomly assigned) and then
>> reloaded.
>> * When a new version of the app is deployed, we'll unpack and start it up to
>> a new folder with a different port. Once the app responds to an HTTP HEAD
>> request (doesn't matter *how* it responds), we'll update the Nginx config
>> file, reload Nginx, send the TERM signal to the old app, and then delete the
>> old app's folder.
>> * When an app is deleted, remove it from the Nginx config file, reload
>> Nginx, send the app the TERM signal, and delete the folder.
>> * We'll automatically store log files for apps (recommendations on good
>> approaches here are welcome).
>> * The deployer itself will provide a web interface to monitor status of apps
>> and view logs.
>> * I have no specific plans for running shell scripts or the like. I believe
>> most of that can be part of the app loading. With the zero-downtime
>> deployment, it's acceptable for the load time to be significant. If anyone
>> has better ideas here, let me know.
>>
>> If we get a good deployment tool going, I think I would next want to set up
>> an AMI (Amazon Machine Image) to make it dead simple to spin up an instance
>> capable of hosting Yesod apps. If we can get it to configure DNS as well,
>> and get a command line tool that automatically performs the EC2 instance
>> creation, I think we'd be golden.
>>
>> Michael