[Haskell-cafe] why does the binary library require so much memory?

[Jeremy Shaw]

Hello,

Is there a work-around? This is killer for Happstack. Most Happstack
applications use IxSet, which in turn uses lists to serialize the data
to/from disk.

Also, why doesn't the stuff get freed eventually?

- jeremy

At Fri, 31 Jul 2009 14:27:30 -0700,
Don Stewart wrote:
> 
> bos:
> > On Fri, Jul 31, 2009 at 1:56 PM, Jeremy Shaw <jeremy at n-heptane.com> wrote:
> > 
> > 
> >     Using encode/decode from Binary seems to permamently increase my
> >     memory consumption by 60x fold. I am wonder if I am doing something
> >     wrong, or if this is an issue with Binary.
> > 
> > 
> > It's an issue with the Binary instance for lists, which forces the entire spine
> > of the list too early. This gives you a gigantic structure to hold onto.
> 
> This is the current instance
> 
>     instance Binary a => Binary [a] where
>         put l  = put (length l) >> mapM_ put l
>         get    = do n <- get :: Get Int
>                     getMany n
> 
>     -- | 'getMany n' get 'n' elements in order, without blowing the stack.
>     getMany :: Binary a => Int -> Get [a]
>     getMany n = go [] n
>      where
>         go xs 0 = return $! reverse xs
>         go xs i = do x <- get
>                      -- we must seq x to avoid stack overflows due to laziness in
>                      -- (>>=)
>                      x `seq` go (x:xs) (i-1)
> 
> It used to be this, though,
> 
>         xs <- replicateM n get     -- now the elems.
> 
> 
> -- Don