[Haskell-cafe] Memory consumption issues under heavy network throughput/concurrency loads

[Ben Bangert]

On Jul 16, 2014, at 6:51 AM, Gregory Collins <greg at gregorycollins.net> wrote:

> OK, I've done some more investigation here, as much time as I can spare for now:
> 	• I'm not sure this program really is leaking forever after all, even on latest GHC. Originally I thought it was, because I was running only 2 pings / client-second as you were. If you increase this to something like 20 pings per client-second, you see the same asymptotics at first but eventually the client plateaus, at least on my machine. I left it running for an hour. The question remains as to why this program exhibits such strange GC behavior (I don't see any reason for it to slowly gobble RAM until plateauing at an arbitrary figure), maybe Simon M can comment.

Yes, with the changes I don't see leaking behavior. As I mentioned in a separate email though, this isn't very useful because the various network libraries in the wild (Warp, Yesod, Websockets, etc) all use ByteStrings for reading frames, etc. on the way to/from the socket. These seem to cause memory fragmentation and issues reclaiming memory, which is the real issue I'm seeing.

Your test has removed the usage of several components that were most likely part of the problem I've had, but which I can't really remove from the fully functioning application as it would require rewriting libraries all the way down the stack.

> 	• The biggest thing you're spending RAM on here is stacks for the threads you create. By default the stack chunk size is 32k, you can lower this with +RTS -kcXX --- using 2kB stacks both programs use <40MB heap resident on my machine. Counting the garbage being generated, the space needed for buffers/etc, and the fact that the binaries themselves are 8MB, I don't think 20kB per active client is unreasonable.
> 	• You can reduce GC pressure somewhat by reusing the output buffer, the "io-streams" branch at my copy of your test repo does this.

Besides for not closing the sockets, your master branch had excellent memory usage. I unfortunately wasn't able to try your io-streams branch as I got this compile error:
https://gist.github.com/bbangert/d26f28b410faaad4e8d2

I'll put together a minimal websocket server that better demonstrates the memory fragmentation issues today.

Cheers,
Ben
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20140716/a2d850e5/attachment.sig>