[Haskell-cafe] why does the binary library require so much memory?
dons at galois.com
Fri Jul 31 17:49:03 EDT 2009
Why don't you use your own instance to serialize IxSet lazily (or
however you would like?)
There's no reason to be constrained to use the [a] instance.
> Is there a work-around? This is killer for Happstack. Most Happstack
> applications use IxSet, which in turn uses lists to serialize the data
> to/from disk.
> Also, why doesn't the stuff get freed eventually?
> - jeremy
> At Fri, 31 Jul 2009 14:27:30 -0700,
> Don Stewart wrote:
> > bos:
> > > On Fri, Jul 31, 2009 at 1:56 PM, Jeremy Shaw <jeremy at n-heptane.com> wrote:
> > >
> > >
> > > Using encode/decode from Binary seems to permamently increase my
> > > memory consumption by 60x fold. I am wonder if I am doing something
> > > wrong, or if this is an issue with Binary.
> > >
> > >
> > > It's an issue with the Binary instance for lists, which forces the entire spine
> > > of the list too early. This gives you a gigantic structure to hold onto.
> > This is the current instance
> > instance Binary a => Binary [a] where
> > put l = put (length l) >> mapM_ put l
> > get = do n <- get :: Get Int
> > getMany n
> > -- | 'getMany n' get 'n' elements in order, without blowing the stack.
> > getMany :: Binary a => Int -> Get [a]
> > getMany n = go  n
> > where
> > go xs 0 = return $! reverse xs
> > go xs i = do x <- get
> > -- we must seq x to avoid stack overflows due to laziness in
> > -- (>>=)
> > x `seq` go (x:xs) (i-1)
> > It used to be this, though,
> > xs <- replicateM n get -- now the elems.
> > -- Don
More information about the Haskell-Cafe