[Haskell-beginners] space leak processing multiple compressed files

[Lorenzo Bolla]

On Tue, Sep 4, 2012 at 11:00 AM, Ian Knopke <ian.knopke at gmail.com> wrote:
> main = do
>
>     -- get a list of file names
>     filelist <- getFileList "testsetdir"
>
>     -- process each compressed file
>     files <- mapM (\x -> do
>                             thisfile <- B.readFile x
>                             return (Z.decompress thisfile)
>                     ) filelist
>
>
>     display $ processEntries files
>
>
>     putStrLn "finished"
>
> -- processEntries
> -- processEntries is defined elsewhere, but basically does some string
> processing per line,
> -- counts the number of resulting elements and sums them per file
> processEntries :: [B.ByteString] -> Int
> processEntries xs = foldl' (\x y -> x + processEntries (B.lines y)) 0 xs

The problem seems to be your `processEntries` function: it is
recursively defined, and as far as I understand, it's never going to
end because "y" (inside the lambda function) is always going to be the
full list of files (xs).

Probably, `processEntries` should be something like:

processEntries = foldl' (\acc fileContent -> acc + processFileContent
fileContent) 0

processFileContent :: B.ByteString -> Int
processFileContent = -- count what you have to, in a file

In fact, processEntries could be rewritten without using foldl':
processEntries = sum . map processFileContent

hth,
L.