[Haskell-cafe] help optimizing memory usage for a program
bulat.ziganshin at gmail.com
Mon Mar 2 12:34:07 EST 2009
Monday, March 2, 2009, 8:16:10 PM, you wrote:
> By the way: I have written the first version of the program to parse
> Netflix training data set in D.
> I also used ncpu * 1.5 threads, to parse files concurrently.
> However execution was *really* slow, due to garbage collection.
> I have also tried to disable garbage collection, and to manually run a
> garbage cycle from time to time (every 200 file parsed), but the
> performance were the same.
may be it will be better to use somewhat like MapReduce and split
your job into 100-file parts which are processed by ncpu concurrently
Bulat mailto:Bulat.Ziganshin at gmail.com
More information about the Haskell-Cafe