[Haskell-beginners] problem with System.Directory.Tree

[Daniel Fischer]

On Monday 07 June 2010 14:06:22, Anand Mitra wrote:
> Hello All,
>
> I want to build a program which will recursively scan a directory and
> build md5sum for all the files. The intent is to do something similar
> to unison but more specific to my requirements. I am having trouble in
> the initial part of building the md5sums.
>
> I did some digging around and found that "System.Directory.Tree" is a
> very close match for what I want to do. In fact after a little poking
> around I could do exactly what I wanted.
>
> ,----
>
> | import Monad
> | import System.Directory.Tree
> | import System.Directory
> | import Data.Digest.Pure.MD5
> | import qualified Data.ByteString.Lazy.Char8 as L
> |
> | calcMD5 =
> |     readDirectoryWith (\x-> liftM md5 (L.readFile x))

Does

calcMD5 =
    readDirectoryWith (\x -> do
        txt <- readFile x
        return $! md5 txt)

help?

>
> `----
>
> This work perfectly for small directories. readDirectoryWith is
> already defined in the library and exactly what we want
>
> ,----
>
> | *Main> calcMD5 "/home/mitra/Desktop/"
> |
> | "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name =
> | "060_LocalMirror_Workflow.t.10.2.62.9.log", file =
> | f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file =
> | 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~",
>
> `----
>
> However when ever I give it something more challenging it gets into
> trouble.
>
> ,----
>
> | *Main> calcMD5 "/home/mitra/laptop/"
> | *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27:
> |    openFile: resource exhausted (Too many open files)
> | *Main> 29~
>
> `----
>
> If I understand what is happening it seems to be doing all the opens
> before consuming them via md5. This works fine for small directories
> but for any practical setup this could potentially be very large. I
> tried forcing the md5 evaluation in the hope that the file descriptor
> will be freed once the entire file is read. That did not help, either
> because I could not get it right or there is some more subtle I am
> missing.
>
> I also had a look at the code in module "System.Directory.Tree" and
> although it gave me some understanding of how it works I am no closer
> to a solution.
>
> regards