[Haskell-cafe] why does the binary library require so much memory?

[Jeremy Shaw]

At Fri, 31 Jul 2009 15:43:49 -0700,
Don Stewart wrote:

> Oh, it is entirely possible to use a different instance, that has
> different semantics for lists.  You want  to write  the list
> incrementally?

I don't think so. In happstack, the idea is to have all your state in
RAM. But, since your machine sometimes goes down, etc, we also want to
back it up to disk so we can restore the state after a reboot. There
has also been talk recently about writing the state to /dev/null after
we restore to make sure it has really been successful decoded. We
don't want to lazily find out that there is undecodable data.

So, the desired experience would be:

 1. A program starts running and populates an IxSet. At this point in
    time n MB of RAM are being used.

 2. We use Binary to snapshot the entire IxSet to disk. Since encode
 outputs an lazy ByteString, I would expect only a modest amount of
 additional memory to be required during this process. After the
 snapshot has been completed, I would expect the app to only be using
 around n MB of RAM.

 3. the application exits and restarts

 4. the application reads in the saved IxSet from disk. Afterwords it
    uses only around n MB of RAM.

Instead, it seems like reading the state in from disk results in, n *
60, the amount of RAM being used. I am not really clear on why. It
seems like the serialized version of the list is going to be *smaller*
than original list, since the original list has all sorts of pointers
and thunks. So, even if decode used a strict ByteString I would expect
less than 2x the memory usage...

Where am I going wrong here?

Thanks!
- jeremy