[Haskell-cafe] binary package: memory problem decoding an IntMap
Manlio Perillo
manlio_perillo at libero.it
Sun Apr 5 16:41:57 EDT 2009
Manlio Perillo ha scritto:
> Hi.
>
> I'm having memory problems decoding a big IntMap.
>
> The data structure is:
>
> IntMap (UArr (Word16 :*: Word8))
>
>
> There are 480189 keys, and a total of 100480507 elements
> (Netflix Prize).
> The size of the encoded (and compressed) data is 184 MB.
>
> When I load data from the Netflix Prize data set, total memory usage is
> 1030 Mb.
>
It seems there is a problem with tuples, too.
I have a:
[(Word16, UArr (Word32 :*:* Word8))]
This eats more memory than it should, since tuples are decoded lazily.
Manlio
More information about the Haskell-Cafe
mailing list