memory slop

[Tim Docker]

On 22/03/11 05:33, Daniel Fischer wrote:
> On Wednesday 23 March 2011 03:32:16, Tim Docker wrote:
>> Is the slop number above likely to be a significant contribution to net
>> memory usage?
>
> Yes, absolutely.
>
>> Are there any obvious reasons why the code below could be
>> generating so much?
>
> I suspect packing a lot of presumably relatively short ByteStrings would
> generate (the lion's share of) the slop. I'm not familiar with the
> internals, though, so I don't know where GHC would put a
> newPinnedByteArray# (which is where your ByteString contents is), what
> alignement requirements those have.

Thanks, I'm aware that that the code could be optimised eg by sticking 
to bytestrings and avoiding Strings and read - there were just to make 
the example simple. I expected this would affect speed, though not 
memory usage.

I'm a bit shocked at the amount of wasted memory here. The sample data 
file has ~61k key/value pair. Hence ~122k ByteStrings - as you point out
many of these are very small (1500 of them are empty). Assuming it's the 
bytestring that are generating slop, I am seeing ~500 bytes on average 
per bytestring!

Tim