[web-devel] Embedded Files for Yesod.Static (was: Content-Length on sendFile)

[Max Cantor]

Changed the subject since we're pretty far from the OP.

I think embedding files is a fantastic idea. 

Also, this would make deployment a bit easier since an entire app can be packaged into a single executable.  

max


On Jun 15, 2011, at 11:20 AM, Michael Snoyman wrote:

> On Wed, Jun 15, 2011 at 5:57 AM, Greg Weber <greg at gregweber.info> wrote:
>> 
>> 
>> On Tue, Jun 14, 2011 at 7:29 PM, Michael Snoyman <michael at snoyman.com>
>> wrote:
>>> 
>>> Let me point out one other distinction: sendFile versus yesod-static.
>>> The former is a function you would call from a normal handler, while
>>> yesod-static is the "magical" package which would actually know all of
>>> the stuff about your files at compile time. For the sendFile case, the
>>> only options for getting the file size are (1) the programmer manually
>>> adding the header and (2) Yesod automatically doing a system call to
>>> get it.
>>> 
>>> As for the behavior of Warp...  while I agree Kazu that we should
>>> reduce system call overhead, it might make sense for Warp to perform
>>> the system call to get file size *if* no content-length header is
>>> present.
>>> 
>>> And I'm still very uncomfortable setting the content-length header for
>>> static files based on compile-time information. In this case, having
>>> the wrong value (e.g., someone modified a CSS file after compiling)
>>> will completely break things and corrupt an open HTTP connection.
>> 
>> Then it should probably not be the default. Lets add a big scary warning to
>> such a setting and tell users the files should be set to read-only
>> permission.
>> The most efficient technique for file serving static files that could be
>> changed would actually be to setup a file notifier (that uses an efficient
>> OS listener, like inotify on linux) that listens to static assets and knows
>> when they are changed or a new one is added and would stat them just when
>> changed. But there would be a race condition if you had to wait for the
>> notification, so you would actually have to have the old file revision on
>> hand- perhaps having a symlink convention for adding new files, but meaning
>> the system could still get screwed up by someone careless.
> 
> I'm really starting to think that the correct solution for this use
> case really is embedded files[1]. It avoids any kind of system call
> overhead, doesn't touch the hard drive at all, and doesn't allow for
> modification of the files after compile time. It would be interesting
> to see if it ends up faster than sendfile.
> 
> Michael
> 
> [1] Using Template Haskell to store the entire static file inside the
> executable during compilation.
> 
>>> 
>>> Michael
>>> 
>>> On Wed, Jun 15, 2011 at 5:22 AM, Kazu Yamamoto <kazu at iij.ad.jp> wrote:
>>>> Greg,
>>>> 
>>>>> I apologize for the confusing terminology. I am not differentiating
>>>>> between
>>>>> sending a static file with sendfile and a streaming response. I
>>>>> am differentiating between 2 different use cases for sending static
>>>>> files
>>>>> (with sendfile). For all of my web applications, I know what all the
>>>>> static
>>>>> files are and they will never change until I deploy another web
>>>>> application.
>>>>> That means I can stat the files once when the application is deployed
>>>>> and keep
>>>>> that information in memory. So I already have the file length
>>>>> information to
>>>>> include in the header, even though I don't do a file stat when the file
>>>>> is
>>>>> requested. wai-app-static and yesod-static supports these techniques.
>>>> 
>>>> Thanks. I think I understand. :)
>>>> 
>>>> So, do you support to *not* change the API (apps should add CL: by
>>>> themselves)?
>>>> 
>>>> --Kazu
>>>> 
>>>> _______________________________________________
>>>> web-devel mailing list
>>>> web-devel at haskell.org
>>>> http://www.haskell.org/mailman/listinfo/web-devel
>>>> 
>>> 
>>> _______________________________________________
>>> web-devel mailing list
>>> web-devel at haskell.org
>>> http://www.haskell.org/mailman/listinfo/web-devel
>> 
>> 
> 
> _______________________________________________
> web-devel mailing list
> web-devel at haskell.org
> http://www.haskell.org/mailman/listinfo/web-devel