[web-devel] Lazy simpleHTTP

[Henning Thielemann]

On Tue, 2 Dec 2008, Paul Brown wrote:

> I'm still a little lost as to why this is necessary and/or desirable.  Some 
> (if not all) of your use cases would be achievable via proper use of HTTP 
> protocol features (100 Continue, Conditional GET, etc.), and it is a question 
> of whether those are well supported by the current implementation.

I do not know, how '100 Continue' would help, since this is for 
conditionally sending the body of the HTTP request, not conditionally 
receiving the body of the HTTP response, right? However, you are right, in 
principle it would help me to use HTTP features, like the Accept header. 
Using it, I could tell, that I only want certain types of data to be send. 
However, e.g. the original HWS ignores the Accept header and when I 
checked other web servers, they do so, as well. So it still seems to be 
the best, to get the header via a GET request and cancel the transfer, 
once I have seen enough of the header.

>  For the laziness in handling response bodies, I think you'd actually 
> want to expose an iteration semantic implemented on top of HTTP chunked 
> encoding and ranged requests, and that would live at a higher level in 
> the stack than the TCP layer.

Yes, I also want lazy download of the body. Certainly an iteratee 
implementation could be cleaner, but probably considerably different from 
the current implementation. I'm afraid 'Range' is supported as bad as 
'Accept' - and what about dynamically generated (maybe even infinite) 
content? So I think I should not bother the web server with the way I want 
to download its content, but just mimic a stupid browser.

My current state can be found here:
    http://code.haskell.org/~thielema/http/
  I do not want to fork and I hope the patches are accepted somewhen and 
nobody adds conflicting patches in the meantime, since this would mean I 
have to re-record all my changes in order to avoid the dreaded exponential 
time darcs conflict resolution. However, currently there are only changes 
behind the scenes and no new visible features. I'm now using monad 
transformers from my explicit-exception package, which simplified a lot of 
the manual exception handling before. I removed some 'reverse's, which are 
a laziness killer, but I'm still not at the end. Chunked transfer is 
particularly difficult with respect to laziness, although I do not know 
why.
  I have added a Monad which further abstracts the Stream concept. It 
allows us to do HTTP communication in absence of IO, which is perfect for 
unit testing and allows lazy processing without unsafeInterleaveIO - you 
only need the unsafeInterleaveIO hidden in hGetContents.