[Haskell-cafe] Memory consumption issues under heavy network throughput/concurrency loads

[Gregory Collins]

On Wed, Jul 16, 2014 at 5:28 PM, Ben Bangert <ben at groovie.org> wrote:

>
> Yes, with the changes I don't see leaking behavior. As I mentioned in a
> separate email though, this isn't very useful because the various network
> libraries in the wild (Warp, Yesod, Websockets, etc) all use ByteStrings
> for reading frames, etc. on the way to/from the socket. These seem to cause
> memory fragmentation and issues reclaiming memory, which is the real issue
> I'm seeing.
>

OK good, that is a data point. BTW the master branch of my version of this
test is still using bytestrings -- the main things that I replaced were the
use of Handle from the high-level "Network" module and the use of async to
run the threads --- not because I thought there was a problem with async
but simply to rule out potentially confounding factors.

Your issue may not be with bytestring fragmentation but that is one of the
working hypothesis and now we are closer to discovering the issue.
Personally I am suspicious that network's Handle API (which, by the way,
there is almost no reason to use --- libraries like io-streams, pipes, and
conduit provide a better user experience) might be the thing that is
leaking RAM. (+johan)

If the issue is GHC pinned heap fragmentation then you can try generating
bytestrings with system malloc instead (which is what we are currently
doing for network reads in io-streams and also IIRC what warp is doing). If
doing that and linking your binary with e.g. tcmalloc fixes the issue, then
I think that would probably be conclusive evidence.


Your test has removed the usage of several components that were most likely
> part of the problem I've had, but which I can't really remove from the
> fully functioning application as it would require rewriting libraries all
> the way down the stack.
>

Of course, but we are trying to find the leak here. Now we can re-add
things to the test, starting with going back to Handle.


>       • The biggest thing you're spending RAM on here is stacks for the
> threads you create. By default the stack chunk size is 32k, you can lower
> this with +RTS -kcXX --- using 2kB stacks both programs use <40MB heap


BTW I lied about this, that parameter controls how much stacks grow if they
overflow, the default stack size for threads is 1kB.


Besides for not closing the sockets, your master branch had excellent
> memory usage. I unfortunately wasn't able to try your io-streams branch as
> I got this compile error:
> https://gist.github.com/bbangert/d26f28b410faaad4e8d2


This thread is just full of fun, isn't it.

G
-- 
Gregory Collins <greg at gregorycollins.net>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.haskell.org/pipermail/haskell-cafe/attachments/20140716/4d16308c/attachment.html>