[Haskell-cafe] why does the binary library require so much memory?

[Bulat Ziganshin]

Hello Jeremy,

Saturday, August 1, 2009, 3:15:02 AM, you wrote:

> So, the desired experience would be:

>  1. A program starts running and populates an IxSet. At this point in
>     time n MB of RAM are being used.

>  2. We use Binary to snapshot the entire IxSet to disk. Since encode
>  outputs an lazy ByteString, I would expect only a modest amount of
>  additional memory to be required during this process.

what really happens here: while you expect that intermediate list
produced lazily (it's haskell, after all!) and immediately stored to
disk in chunks, "length l" in Binary instance forces entire list to be
calculated before producing any output

it should be easy to fix: if your IxSet have builtin elements counter
or way to measure it without allocating tons of memory, you just need
to make something like the following instance:

    instance Binary a => Binary (IxSet a) where
        put l  = put (IxSet.length l) >> mapM_ put l
        get    = fmap list_to_IxSet get


-- 
Best regards,
 Bulat                            mailto:Bulat.Ziganshin at gmail.com