[Haskell-cafe] Program using 500MB RAM to process 5MB file

[Luke Palmer]

2009/4/2 <lucas at die.net.au>

> I'm relatively new to haskell so as one does, I am rewriting an
> existing program in haskell to help learn the language.
>
> However, it eats up all my RAM whenever I run the program.
>
> http://hpaste.org/fastcgi/hpaste.fcgi/view?id=3175#a3175
>
> Obviously I'm doing something wrong, but without my magical FP pants I
> don't know what that might be.


(1) You are using plain Strings.  Those things are like 8 bytes per
character (or something, someone more knowledgeable can give a more accurate
figure).  Use bytestrings (with bytestring-utf8 if you need it) instead.

(2) You are parsing strictly, meaning you have to read the whole input file
before anything can be output.  This may be necessary for your application,
but Haskell is *very* strong with streaming applications.  Change to a lazy
parser and you will run in constant memory.

(I don't know offhand of any lazy parsing libraries, but I've heard them
discussed before, so they're somewhere)

Luke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org/pipermail/haskell-cafe/attachments/20090402/8641e772/attachment-0001.htm