[Haskell-cafe] memory-efficient data type for Netflix data -
UArray Int Int vs UArray Int Word8
manlio_perillo at libero.it
Sun Mar 1 09:03:14 EST 2009
Kenneth Hoste ha scritto:
> I'm having a go at the Netflix Prize using Haskell. Yes, I'm brave.
> I kind of have an algorithm in mind that I want to implement using Haskell,
> but up until now, the main issue has been to find a way to efficiently
> the data...
> For people who are not familiar with the Netflix data, in short, it
> consist of
> roughly 100M (1e8) user ratings (1-5, integer) for 17,770 different
> movies, coming from
> 480,109 different users.
I have written a simple program that parses the Netflix training data
set, using this data structure:
type MovieRatings = IntMap (UArr Word32, UArr Word8)
The ratings are grouped by movies.
The parsing is done in:
On a DELL Inspiron 6400 notebook,
Intel Core2 T7200 @ 2.00GHz, and 2 GB memory.
However the memory used is about 1.4 GB.
How did you manage to get 700 MB memory usage?
Note that the minimum space required is about 480 MB (assuming 4 byte
integer for the ID, and 1 byte integer for rating).
Using a 4 byte integer for both ID and rating, the space required is
about 765 MB.
1.5 GB is the space required if one uses a total of 16 bytes to store
both the ID and the rating.
Maybe it is the garbage collector that does not release memory to the
Thanks Manlio Perillo
More information about the Haskell-Cafe