[Haskell-cafe] ST not strict enough?
jason.dusek at gmail.com
Wed Nov 16 19:39:39 CET 2011
2011/11/15 Roman Cheplyaka <roma at ro-che.info>:
> * Jason Dusek <jason.dusek at gmail.com> [2011-11-15 20:08:48+0000]
>> I'm having some trouble with memory usage in rebuilding a
>> ByteString with some sequences escaped. I thought I'd try
>> vectors. However, it seems that even a relatively simple
>> function, one that places all the bytes of a ByteString in to a
>> vector, uses a great deal of memory.
> I think what's happening here is ByteString's "strictness" makes things
> actually lazy on your side.
> Namely, unpack function produces its result "strictly", whole list at
> once. As a result, the resulting list cannot be consumed one-by-one,
> so it takes memory. You see ST thunks because
> mapM_ f as = sequence_ (map f as)
> and that map probably gets fused with unpack.
> I guess the proper solution here is to use lazy bytestring and make sure
> the chunks are not very big.
Switching to the lazy ByteStrings API does, indeed, help; total
memory usage is around 16M. I will have a look at the rules that
are fired to see what I can learn.
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
More information about the Haskell-Cafe