[Haskell-cafe] Re: Processing of large files

Peter Simons simons at cryp.to
Tue Nov 2 10:56:45 EST 2004

John Goerzen writes:

 >> Read and process the file in blocks:

 > I don't think that would really save much memory [...]

Given that the block-oriented approach has constant space
requirements, I am fairly confident it would save memory.

 > and in fact, would likely just make the code a lot more
 > complex. It seems like a simple wrapper around
 > hGetContents over a file that uses block buffering would
 > suffice.

Either your algorithm can process the input in blocks or it
cannot. If it can, it doesn't make one bit a difference if
you do I/O in blocks, because your algorithm processes
blocks anyway. If your algorithm is *not* capable of
processing blocks, then don't bother with block-oriented I/O
-- and don't process large files unless you have 'seq' in
all the right places. I happen to find adding those 'seq'
calls much more difficult to get right than writing a
computation that can be restarted. But your mileage may


More information about the Haskell-Cafe mailing list