[Haskell-cafe] broken IO support in uvector package, when using
non primitive types
manlio_perillo at libero.it
Sat Mar 14 12:38:45 EDT 2009
Daniel Peebles ha scritto:
> I have added UAProd-based Binary instances to my copy of the uvector
> repo at http://patch-tag.com/publicrepos/pumpkin-uvector .
I can confirm that it works for me.
However I have now a memory problem with data decoding.
I need to serialize the Netflix Prize training dataset.
When I parse the data from original data set, memory usage is about 640
But when I load the data serialized and compressed, (as [UArr (Word32
*:* Word8)]), memory usage is about 840 MB...
The culprit is probably the decoding of the list (17770 elements).
 I have written a script in Python that parse the data, and it only
takes 491 MB
(using a list of a tuple with two compact arrays from numpy).
So, GHC has memory problems here.
Thanks Manlio Perillo
More information about the Haskell-Cafe