[Haskell-cafe] Program using 500MB RAM to process 5MB file

[Malcolm Wallace]

> > (2) You are parsing strictly, meaning you have to read the whole  
> > input file before anything can be output.

This is likely the main performance problem.  I'm guessing you are using
parsec.  Try switching to polyparse if you want to try out lazy parser
combinators instead.  (module Text.ParserCombinators.Poly.Lazy)
    http://www.cs.york.ac.uk/fp/polyparse/

There are other incremental parsers out there too, although they may be
more complicated because they solve a larger problem:
    http://yi-editor.blogspot.com/2008/11/incremental-parsing-in-yi.html

> I also suspect that manyTill is a really bad choice, since it doesn't 
> give you anything until the end token.

This is not an "also" - it is the same problem of too much strictness.
Polyparse's combinator 'manyFinally' is the corresponding combinator
that works lazily if you want it to.

Regards,
    Malcolm