[Haskell-cafe] binary package: memory problem decoding an IntMap
nicolas.pouillard at gmail.com
Sun Apr 5 17:10:27 EDT 2009
Excerpts from Manlio Perillo's message of Sun Apr 05 22:41:57 +0200 2009:
> Manlio Perillo ha scritto:
> > Hi.
> > I'm having memory problems decoding a big IntMap.
> > The data structure is:
> > IntMap (UArr (Word16 :*: Word8))
> > There are 480189 keys, and a total of 100480507 elements
> > (Netflix Prize).
> > The size of the encoded (and compressed) data is 184 MB.
> > When I load data from the Netflix Prize data set, total memory usage is
> > 1030 Mb.
> It seems there is a problem with tuples, too.
> I have a:
> [(Word16, UArr (Word32 :*:* Word8))]
> This eats more memory than it should, since tuples are decoded lazily.
Why not switch to [(Word16 :*: UArr (Word32 :*: Word8))] then?
More information about the Haskell-Cafe