[Haskell-cafe] Memory consumption issues under heavy network throughput/concurrency loads

[Ben Bangert]

On Jul 17, 2014, at 11:56 AM, Simon Marlow <marlowsd at gmail.com> wrote:

> You guys seem to be doing a good job of narrowing this down.  We know there are issues with fragmentation when using ByteStrings; in the worst case each ByteString can pin up to its size rounded up to 4Kbytes.  If you're not keeping old ByteStrings around but are regularly recycling them, then you shouldn't see this problem.  I believe we made some improvements (allegedly) in 7.6 to the fragmentation behaviour.
> 
> Do let me know when you've narrowed it down to something you think I should look at.

I made a slightly more complex version of the echo server that uses attoparsec to read into a basic datatype here:
https://gist.github.com/bbangert/592e3dcc0253f275e9a3

I've been unable to make it leak using the socket handling code Gregory supplied, and its growth bounds seem fine (113Mb for 5k connections each sending 2 pings/sec). As I flesh out the HTTP/2 frame handling (which will result in substantial increases of temporary ByteStrings), I'll see if any fragmentation issues return.

Thanks!
Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20140720/f33d9e07/attachment.sig>