[Haskell-cafe] Stack space overflow in HaskellNet

[Donn Cave]

Quoth Manfred Lotz <manfred.lotz at arcor.de>,
...
> The problem seems to lie in the HaskellNet package. If for example I
> only fetch a specific message 
>    m <- fetch con 2092
> having a size of some 1.2m then I get the same stack overflow.
>
> If at runtime I specify +RTS -K40M -RTS it works but takes over 40
> seconds to fetch the message. 

That's not so good, but I wouldn't be surprised if it's a natural parsing
problem, I mean it's just a lot of data to run through a Haskell parser.

IMAP does give you the means to mitigate the problem - the big data
transfer in a FETCH response is preceded by a byte count - but to really
take advantage of that, how far do you go?  I don't have much experience
with general purpose parsers, do they often support an efficient counted
string read?  Is it OK to return String, or do we need a more efficient
type (e.g., ByteString?)  Is it OK to return any kind of string value -
given that a message part could be arbitrarily long (and needs to be
decoded), do you go to a lot of trouble to support large message parts
but not extremely large ones?

For me, the answer is for the parser to bail out, reporting the counted
input as a count but leaving it to the application to actually effect
the data transfer and return to finish the parse.  That's only moderately
complicated, but it's part of a general philosophy about application
driven I/O vs. protocol parsing that seems to be mine alone.

I have no idea how much could be done to tighten up HaskellNet.IMAP.
Someone who understands it well enough might be able to get a miraculous
improvement with a strictness annotation or something.  Maybe you
could track that down with profiling.

	Donn