[Haskell-cafe] Memory consumption issues under heavy network throughput/concurrency loads

[Simon Marlow]

You guys seem to be doing a good job of narrowing this down.  We know 
there are issues with fragmentation when using ByteStrings; in the worst 
case each ByteString can pin up to its size rounded up to 4Kbytes.  If 
you're not keeping old ByteStrings around but are regularly recycling 
them, then you shouldn't see this problem.  I believe we made some 
improvements (allegedly) in 7.6 to the fragmentation behaviour.

Do let me know when you've narrowed it down to something you think I 
should look at.

Cheers,
Simon

On 16/07/2014 14:51, Gregory Collins wrote:
> OK, I've done some more investigation here, as much time as I can spare
> for now:
>
>   * I'm not sure this program really is leaking forever after all, even
>     on latest GHC. Originally I thought it was, because I was running
>     only 2 pings / client-second as you were. If you increase this to
>     something like 20 pings per client-second, you see the same
>     asymptotics at first but eventually the client plateaus, at least on
>     my machine. I left it running for an hour. The question remains as
>     to why this program exhibits such strange GC behavior (I don't see
>     any reason for it to slowly gobble RAM until plateauing at an
>     arbitrary figure), maybe Simon M can comment.
>   * The biggest thing you're spending RAM on here is stacks for the
>     threads you create. By default the stack chunk size is 32k, you can
>     lower this with +RTS -kcXX --- using 2kB stacks both programs use
>     <40MB heap resident on my machine. Counting the garbage being
>     generated, the space needed for buffers/etc, and the fact that the
>     binaries themselves are 8MB, I don't think 20kB per active client is
>     unreasonable.
>   * You can reduce GC pressure somewhat by reusing the output buffer,
>     the "io-streams" branch at my copy of your test repo does this.
>
> G
>
>
> On Wed, Jul 16, 2014 at 1:00 PM, Gregory Collins
> <greg at gregorycollins.net <mailto:greg at gregorycollins.net>> wrote:
>
>
>     On Wed, Jul 16, 2014 at 5:46 AM, Ben Bangert <ben at groovie.org
>     <mailto:ben at groovie.org>> wrote:
>
>         I've run the new server code, indeed it initially takes less
>         memory. However, when I ctrl-C the testing client, it fails to
>         properly close sockets now, they stay open forever (the original
>         code I posted always closed the sockets when I killed the
>         testing client). Did you try ctrl-c the test client and
>         re-running it several times?
>
>
>     Yes once I exhibited the leaking behaviour I stopped (and didn't
>     implement cleanup properly). It should be ok now.
>
>
>         I'm guessing that the recv call is blocking forever and failing
>         somehow to note that the socket was actually closed under it,
>         while the hGetline I was using properly detects this condition
>         and closes the socket when it notices the other side has left.
>
>
>     The Handle stuff will still end up calling recv() under the hood.
>
>
>         I should note that when asking for even 1000 echo clients, the
>         test client as you've changed it dies when I try to launch it
>         (it shows Clients Connected: 0, then just exits silently).
>
>
>     Are you running out of FDs?
>
>         I had to use my old testing client to test the server changes
>         you made. If the sockets would properly close when terminating
>         the client abruptly, then its quite possible memory usage would
>         remain lower as it definitely took much much less memory for the
>         first 2k echo clients.
>
>
>     I've also updated my copy of the the test code to work with the GHC
>     7.4.1 that's installed on my Ubuntu box at work (64-bit) --- and I
>     *don't* see the client leak there, resident heap plateaus at around
>     90MB.
>
>     G
>     --
>     Gregory Collins <greg at gregorycollins.net
>     <mailto:greg at gregorycollins.net>>
>
>
>
>
> --
> Gregory Collins <greg at gregorycollins.net <mailto:greg at gregorycollins.net>>