[Haskell-cafe] Strange memory consumption problems in something that should be tail-recursive

Duncan Coutts duncan.coutts at worc.ox.ac.uk
Tue Feb 13 15:59:18 EST 2007

On Tue, 2007-02-13 at 15:27 -0500, Jefferson Heard wrote:
> Hi, I am running the following code against a 210 MB file in an attempt to 
> determine whether I should use alex or whether, since my needs are very 
> performance oriented, I should write a lexer of my own.  I thought that 
> everything I'd written here was tail-recursive

Isn't that exactly the problem - that it's tail recursive? You do not
want it to be tail recursive since then it must consume the whole input
before producing any output. You want it to be as lazy as possible so
that it can start producing tokens as soon as possible without having to
consume everything.

If performance is really important to you then you may also want to
investigate lexing from a lazy ByteString. Alex can now do that (darcs
version) or you can do it by hand as you're trying now.


More information about the Haskell-Cafe mailing list