[Haskell-cafe] help optimizing memory usage for a program

[Bulat Ziganshin]

Hello Manlio,

Monday, March 2, 2009, 8:16:10 PM, you wrote:

> By the way: I have written the first version of the program to parse
> Netflix training data set in D.
> I also used ncpu * 1.5 threads, to parse files concurrently.

> However execution was *really* slow, due to garbage collection.
> I have also tried to disable garbage collection, and to manually run a
> garbage cycle from time to time (every 200 file parsed), but the 
> performance were the same.

may be it will be better to use somewhat like MapReduce and split
your job into 100-file parts which are processed by ncpu concurrently
executed scripts?


-- 
Best regards,
 Bulat                            mailto:Bulat.Ziganshin at gmail.com