[Haskell-cafe] binary package: memory problem decoding an IntMap

[Nicolas Pouillard]

Excerpts from Manlio Perillo's message of Sun Apr 05 22:41:57 +0200 2009:
> Manlio Perillo ha scritto:
> > Hi.
> > 
> > I'm having memory problems decoding a big IntMap.
> > 
> > The data structure is:
> > 
> > IntMap (UArr (Word16 :*: Word8))
> > 
> > 
> > There are 480189 keys, and a total of 100480507 elements
> > (Netflix Prize).
> > The size of the encoded (and compressed) data is 184 MB.
> > 
> > When I load data from the Netflix Prize data set, total memory usage is
> > 1030 Mb.
> > 
> 
> It seems there is a problem with tuples, too.
> 
> I have a:
>      [(Word16, UArr (Word32 :*:* Word8))]
> 
> This eats more memory than it should, since tuples are decoded lazily.

Why not switch to [(Word16 :*: UArr (Word32 :*: Word8))] then?

-- 
Nicolas Pouillard