[Haskell-cafe] why does the binary library require so much memory?

[Don Stewart]

Why don't you use your own instance to serialize IxSet lazily (or
however you would like?)

There's no reason to be constrained to use the [a] instance.

-- Don

jeremy:
> Hello,
> 
> Is there a work-around? This is killer for Happstack. Most Happstack
> applications use IxSet, which in turn uses lists to serialize the data
> to/from disk.
> 
> Also, why doesn't the stuff get freed eventually?
> 
> - jeremy
> 
> At Fri, 31 Jul 2009 14:27:30 -0700,
> Don Stewart wrote:
> > 
> > bos:
> > > On Fri, Jul 31, 2009 at 1:56 PM, Jeremy Shaw <jeremy at n-heptane.com> wrote:
> > > 
> > > 
> > >     Using encode/decode from Binary seems to permamently increase my
> > >     memory consumption by 60x fold. I am wonder if I am doing something
> > >     wrong, or if this is an issue with Binary.
> > > 
> > > 
> > > It's an issue with the Binary instance for lists, which forces the entire spine
> > > of the list too early. This gives you a gigantic structure to hold onto.
> > 
> > This is the current instance
> > 
> >     instance Binary a => Binary [a] where
> >         put l  = put (length l) >> mapM_ put l
> >         get    = do n <- get :: Get Int
> >                     getMany n
> > 
> >     -- | 'getMany n' get 'n' elements in order, without blowing the stack.
> >     getMany :: Binary a => Int -> Get [a]
> >     getMany n = go [] n
> >      where
> >         go xs 0 = return $! reverse xs
> >         go xs i = do x <- get
> >                      -- we must seq x to avoid stack overflows due to laziness in
> >                      -- (>>=)
> >                      x `seq` go (x:xs) (i-1)
> > 
> > It used to be this, though,
> > 
> >         xs <- replicateM n get     -- now the elems.
> > 
> > 
> > -- Don