[Haskell-cafe] Debugging ByteString and Data.Binary.Get memory usage

Kyle Hanson hanooter at gmail.com
Fri Aug 30 06:33:52 CEST 2013

Thanks Bob,

I made it foldr because it was meant to simulate the sequential IO action
that my server uses to populate the Map.

I found the problem to be that I need to force the map to evaluate so
adding a little $! fixed the problem

Kyle Hanson

On Thu, Aug 29, 2013 at 9:09 PM, Bob Ippolito <bob at redivi.com> wrote:

> Building a map with foldr seems unwise, have you tried doing it with
> fromListWith instead? Or foldl'? In either case, since you don't even put
> the map into WHNF, none of the computation is done at all in either case
> until the first lookup.
> On Thu, Aug 29, 2013 at 3:35 PM, Kyle Hanson <hanooter at gmail.com> wrote:
>> OK
>> I have a bunch of BSON documents that I convert to ByteStrings, put in a
>> Map, and write to a socket based on the response. I noticed some high
>> memory usage (in the GBs) so I decided to investigate. I simplified my
>> problem into a small program that demonstrates clearer what is happening.
>> I wrote two versions, one with a Lazy Map and Lazy ByteStrings and one
>> with a Strict Map and Strict ByteStrings. Both share the same memory
>> behavior (except the lazy BS one is faster)
>> Here is the strict version:
>> http://lpaste.net/92298
>> And here is the lazy version:
>> http://lpaste.net/92299
>> I wrote this and compared the memory and speed behavior of ByteStrings
>> generated by converting it from a BSON document and ByteStrings generated
>> more purely.
>> The length of the ByteString from a BSON document is 68k and the length
>> of the "pure" BS is 70k.
>> This is my weird memory behavior, both BSON and "pure" methods use the
>> same amount of memory after inserting 10k of them (90mb)
>> However when I go to lookup a value, the BSON Map explodes the memory to
>> over 250mb. Even if I lookup just 1 value. Looking up any number of values
>> in the "pure BS" keeps the memory usage stable (90mb).
>> I am hoping someone can help me understand this. I have read some posts
>> about Temporary ByteStrings causing memory issues but I don't know how to
>> get started debugging.
>> --
>> Kyle Hanson
>> _______________________________________________
>> Haskell-Cafe mailing list
>> Haskell-Cafe at haskell.org
>> http://www.haskell.org/mailman/listinfo/haskell-cafe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20130829/18f91d47/attachment.htm>

More information about the Haskell-Cafe mailing list