[Haskell-cafe] Processing of large files

[Ketil Malde]

"Alexander N. Kogan" <alexander at kogan.nnov.ru> writes:

> How should I modify it to make it useful on large file?
> It eats too much memory...

> procFile =
>     putStrLn       .
>     show       .
>     foldl merge []      .
      ^^^^^

>     words

foldl is infamous for building the complete list, before evaluating
anything. Did you try foldr or foldl' instead?   Also, you may want to
make 'merge' more strict; I suspect you build lazy tuples that look like
("word",1+1+1+1+...) and only get evaluated at the end.

Using heap profiling will probably give you some hints (well
documented in the GHC manual, ask if you get stuck)

-kzm
-- 
If I haven't seen further, it is by standing in the footprints of giants