[Haskell-cafe] Debugging ByteString and Data.Binary.Get memory usage

[Kyle Hanson]

Thanks Bob,

I made it foldr because it was meant to simulate the sequential IO action
that my server uses to populate the Map.

I found the problem to be that I need to force the map to evaluate so
adding a little $! fixed the problem

--
Kyle Hanson




On Thu, Aug 29, 2013 at 9:09 PM, Bob Ippolito <bob at redivi.com> wrote:

> Building a map with foldr seems unwise, have you tried doing it with
> fromListWith instead? Or foldl'? In either case, since you don't even put
> the map into WHNF, none of the computation is done at all in either case
> until the first lookup.
>
>
> On Thu, Aug 29, 2013 at 3:35 PM, Kyle Hanson <hanooter at gmail.com> wrote:
>
>> OK
>>
>> I have a bunch of BSON documents that I convert to ByteStrings, put in a
>> Map, and write to a socket based on the response. I noticed some high
>> memory usage (in the GBs) so I decided to investigate. I simplified my
>> problem into a small program that demonstrates clearer what is happening.
>>
>> I wrote two versions, one with a Lazy Map and Lazy ByteStrings and one
>> with a Strict Map and Strict ByteStrings. Both share the same memory
>> behavior (except the lazy BS one is faster)
>>
>> Here is the strict version:
>>
>> http://lpaste.net/92298
>>
>> And here is the lazy version:
>>
>> http://lpaste.net/92299
>>
>> I wrote this and compared the memory and speed behavior of ByteStrings
>> generated by converting it from a BSON document and ByteStrings generated
>> more purely.
>>
>> The length of the ByteString from a BSON document is 68k and the length
>> of the "pure" BS is 70k.
>>
>> This is my weird memory behavior, both BSON and "pure" methods use the
>> same amount of memory after inserting 10k of them (90mb)
>>
>> However when I go to lookup a value, the BSON Map explodes the memory to
>> over 250mb. Even if I lookup just 1 value. Looking up any number of values
>> in the "pure BS" keeps the memory usage stable (90mb).
>>
>> I am hoping someone can help me understand this. I have read some posts
>> about Temporary ByteStrings causing memory issues but I don't know how to
>> get started debugging.
>>
>> --
>> Kyle Hanson
>>
>> _______________________________________________
>> Haskell-Cafe mailing list
>> Haskell-Cafe at haskell.org
>> http://www.haskell.org/mailman/listinfo/haskell-cafe
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20130829/18f91d47/attachment.htm>