[Haskell-beginners] problem with System.Directory.Tree

[Anand Mitra]

Hello All,

I want to build a program which will recursively scan a directory and
build md5sum for all the files. The intent is to do something similar
to unison but more specific to my requirements. I am having trouble in
the initial part of building the md5sums.

I did some digging around and found that "System.Directory.Tree" is a
very close match for what I want to do. In fact after a little poking
around I could do exactly what I wanted.

,----
| import Monad
| import System.Directory.Tree
| import System.Directory
| import Data.Digest.Pure.MD5
| import qualified Data.ByteString.Lazy.Char8 as L
|
| calcMD5 =
|     readDirectoryWith (\x-> liftM md5 (L.readFile x))
`----

This work perfectly for small directories. readDirectoryWith is
already defined in the library and exactly what we want

,----
| *Main> calcMD5 "/home/mitra/Desktop/"
|
| "/home/mitra" :/ Dir {name = "Desktop", contents = [File {name =
| "060_LocalMirror_Workflow.t.10.2.62.9.log", file =
| f687ad04bc64674134e55c9d2a06902a},File {name = "cmd_run", file =
| 6f334f302b5c0d2028adeff81bf2a0d9},File {name = "cmd_run~",
`----

However when ever I give it something more challenging it gets into
trouble.

,----
| *Main> calcMD5 "/home/mitra/laptop/"
| *** Exception: /home/mitra/laptop/ell/calc-2.02f/calc.info-27:
|    openFile: resource exhausted (Too many open files)
| *Main> 29~
`----

If I understand what is happening it seems to be doing all the opens
before consuming them via md5. This works fine for small directories
but for any practical setup this could potentially be very large. I
tried forcing the md5 evaluation in the hope that the file descriptor
will be freed once the entire file is read. That did not help, either
because I could not get it right or there is some more subtle I am
missing.

I also had a look at the code in module "System.Directory.Tree" and
although it gave me some understanding of how it works I am no closer
to a solution.

regards
-- 
Anand Mitra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.haskell.org/pipermail/beginners/attachments/20100607/28ee5ff9/attachment.html