[Haskell-cafe] help optimizing memory usage for a program

[Manlio Perillo]

Hi.

After some work I have managed to implement two simple programs that 
parse the Netflix Prize data set.

For details about the Netflix Prize, there was a post by Kenneth Hoste 
some time ago.

I have cabalized the program, and made available here:
http://haskell.mperillo.ath.cx/netflix-0.0.1.tar.gz

The process-data-1 program parse the training data set, grouping ratings 
by movies.
The process-data-2 program, instead, groups ratings by users.


The structure of the two programs is similar: I create singletons IntMap 
and then use foldl + union to merge them together.


The data structure used is:

type Rating = Word32 :*: Word8 -- id, rating
type MovieRatings = IntMap (UArr Rating)

UArr is from uvector package.


The training data set contains about 100 million ratings, so, in theory, 
the amount of memory required to store pairs of (id, rating) is about 
480 MB.


The process-data-1 program parse the entire dataset using about 1.4 GB 
of memory (3x increment).

This is strange.
The memory required is proportional to the number of ratings.
It may be IntMap the culprit, or the garbage collector than does not 
release the memory to the operating system (or, worse, does not 
deallocate all used temporary memory).


The process-data-2 has much more problems.
When I try to parse *only* 500 movie ratings, the memory usage is
471 MB; 780 MB after all data has been fully evaluated (array 
concatenations).


I'm using ghc 6.8.2 on Debian Etch.
I would like to do some tests with recent GHC versions, but it may be a 
problem.



Thanks  Manlio Perillo