[web-devel] [WAI] Switching to WAI/Warp...

[Michael Snoyman]

On Sun, Feb 20, 2011 at 12:00 PM, Bardur Arantsson <spam at scientician.net> wrote:
> On 2011-02-20 10:29, Michael Snoyman wrote:
>>
>> On Sun, Feb 20, 2011 at 11:04 AM, Bardur Arantsson<spam at scientician.net>
>>  wrote:
>
> [--snip--]
>>>
>>> Is there any way to avoid this extra "fromByteString"?
>>
>> blaze-builder is usually pretty intelligent about this. If I remember
>> correctly, Simon Meier said that it won't do a memory copy for
>> ByteStrings larger than 8k. In any event, if you want to force
>> insertion instead of copying, replace fromByteString with
>> insertByteString. It *might* be that the CPU overhead is warranted,
>> however: you may end up seeing increased system call overhead with
>> this switch, since the average chunk size will be smaller.
>>
>
> Good point; I just tried upping the chunk size to 32K and explicitly using
> insertByteString. Even with those changes I'm still seeing a lot of CPU
> usage (5-10%).
>
> I should say that I'm using a version of the enumFile enumerator from
> Data.Enumerator.Binary that I've adapted to support byte ranges. It may be
> the case that I've just done something horribly stupid or inefficient. I've
> attached the code for the enumerator.

I don't see any problems with your implementation, but John Millikin
is definitely the guy to speak to about that. He will be able to give
you a more definitive answer than I.

> I guess I'll have to try to get some profiling data to see where the time is
> actually being spent. I suppose it's about time I learned a bit about
> profiling my Haskell code :).

It's entirely possible that WAI/enumerators/builder/Warp is adding
some overhead. But what exactly are you comparing against? Warp is
*definitely* doing some extra stuff that a simple Data.ByteString.hPut
is not, such as timeout handling. I'd be interested in any numbers
that you come up with from profiling, please do share.

> [--snip bits about enumerators--]
>
> Thanks for the explanation. For some reason I'm having a little trouble
> "connecting" all the type signatures for enumerator/iterator/iteratee.
> Hopefully it'll get better with practice.

There's no question that it's a difficult concept to get started with,
but it many ways it's like learning monads: the concept looks
ridiculously complicated at first, then you think that it's just like
(spaceships/nuclear waste/burritos), and then it finally *really*
clicks and becomes second nature. The only real solution is what
you've already said: practice.

If you haven't seen it already, I wrote a three-part series on the
enumerator package. I've moved that package into a chapter of the
Yesod book[1] so you can read it all on a single page if you like.
Hope it helps!

Michael

[1] http://docs.yesodweb.com/book/enumerator