[web-devel] questions about ResponseEnumerator

[Michael Snoyman]

On Sun, Oct 23, 2011 at 5:09 PM, Gregory Collins
<greg at gregorycollins.net> wrote:
> On Sat, Oct 22, 2011 at 10:20 PM, Michael Snoyman <michael at snoyman.com> wrote:
>>
>> I think Greg's/Snap's approach of a separate timeout for the status
>> and headers is right on the money. It should never take more than one
>> timeout cycle to receive a full set of headers, regardless of how slow
>> the user's connection, and given a reasonable timeout setting from the
>> user (anything over 2 seconds should be fine I'd guess, and our
>> default is 30 seconds).
>
> That's fairly uncontroversial.
>
>
>> The bigger question is what we do about the request body. A simple
>> approach might just be that if we receive a packet from the client
>> which is less than a certain size (user defined, maybe 2048 bytes is a
>> good default) it does not tickle the timeout at all. Obviously this
>> means a malicious program could be devised to send precisely 2048
>> bytes per timeout cycle... but I don't think there's any way to do
>> better than this.
>
> This doesn't really work either. I've already posted code in this
> thread for what I think is the only reasonable option, which is rate
> limiting. The way we've implemented rate limiting is:
>
>   1) any individual data packet must arrive within N seconds (the
> usual timeout)
>
>   2) when you receive a packet, you compute the data rate in bytes
> per second -- if it's lower than X bytes/sec (where X is a policy
> decision left up to the user), the connection is killed
>
>   3) the logic from 2) only kicks in after Y seconds, to cover cases
> where the client needs to do some expensive initial setup. Y is also a
> policy decision.

As you mention below, the approach I outline is fairly close to rate
limiting. I would argue it's *more* correct. In the rate limiting
case, a client could send a burst of a huge amount of data at the
beginning of the connection, and then send single bytes every 20
seconds for the next few minutes, and rate limiting would allow it to
happen. The approach below (call it minimum packet size?) wouldn't.

Additionally, I think minimum packet size can be implemented much more
efficiently, as there is significantly less data to track.

>> We *have* to err on the side of allowing attacks, otherwise we'll end up with disconnecting valid requests.
>
> I don't agree with this. Some kinds of "valid" requests are
> indistinguishable from attacks. You need to decide what's more
> important: letting some guy on a 30-kilobit packet radio connection
> upload a big file, or letting someone DoS your server.

Fair enough, let me rephrase: there's a gray area between attack and
valid request, and by default I'd like to draw the line so as to
incorporate most of the gray zone.

>
>
>> In other words, here's how I'd see the timeout code working:
>>
>> 1. A timeout is created at the beginning of a connection, and not
>> tickled at all until all the request headers are read in.
>> 2. Every time X (default: 2048) bytes of the request body are read,
>> the timeout is tickled.
>
> Note that this is basically a crude form of rate-limiting (at X/T
> bytes per second). Why not do it "properly"?
>
> G
> --
> Gregory Collins <greg at gregorycollins.net>
>