[Haskell-beginners] Reading Multiple Files and Iterate Function Application

[Daniel Fischer]

On Tuesday 12 October 2010 11:02:47, Lorenzo Isella wrote:
> Thanks Thomas. Yep, I do need some extra reading unfortunately.
> One question: if I was to apply a function on many files file1,
> file2...using e.g. Python, this would be my pipeline
> read file1
> do stuff on file 1
>
> read file2
> do stuff on file 2
>
> ......
>
> Now, due to the laziness of haskell, can I here resort to this approach
>
> read file1, file2... into a single list
>
> map (do-my-stuff) on list
>
> As far as I understand, this should not result e.g. into a huge RAM
> consumptions since files are read and processed only when needed (hence
> one at the time).
> Am I on the right track?

Yes, but there are dangers on that way.
With readFile, the contents are read lazily upon demand, but the file is 
opened immediately for reading. So

contentsList <- mapM readFile fileList

or

allContents <- fmap concat $ mapM readFile fileList

can make you run out of file handles if fileList is long enough.

Also, the file handles aren't closed until the entire contents of the file 
has been read (there are a few situations where the handle is closed 
earlier) and they're not guaranteed to be immediately closed when the end 
of the file has been reached, they could linger for a GC or two.
That means you can also run out of file handles when you process the files 
sequentially (if you have a bad consumption pattern).

The memory usage depends on your consumption pattern, independent of 
whether theSting[s] you process come[s] from file readings or from a non-IO 
generator.
If you keep references to the beginning of the list, you get a leak, if you 
consume the list sequentially, it runs in small space.

> Cheers
>
> Lorenzo