twd2 at dockerz.net
Wed Mar 23 05:00:29 CET 2011
On 22/03/11 05:33, Daniel Fischer wrote:
> On Wednesday 23 March 2011 03:32:16, Tim Docker wrote:
>> Is the slop number above likely to be a significant contribution to net
>> memory usage?
> Yes, absolutely.
>> Are there any obvious reasons why the code below could be
>> generating so much?
> I suspect packing a lot of presumably relatively short ByteStrings would
> generate (the lion's share of) the slop. I'm not familiar with the
> internals, though, so I don't know where GHC would put a
> newPinnedByteArray# (which is where your ByteString contents is), what
> alignement requirements those have.
Thanks, I'm aware that that the code could be optimised eg by sticking
to bytestrings and avoiding Strings and read - there were just to make
the example simple. I expected this would affect speed, though not
I'm a bit shocked at the amount of wasted memory here. The sample data
file has ~61k key/value pair. Hence ~122k ByteStrings - as you point out
many of these are very small (1500 of them are empty). Assuming it's the
bytestring that are generating slop, I am seeing ~500 bytes on average
More information about the Glasgow-haskell-users