[Haskell-cafe] Amazonka, conduit and sockets not closing

[Magnus Therning]

Viktor Dukhovni <ietf-dane at dukhovni.org> writes:

> On Thu, Nov 26, 2020 at 02:12:49PM +0100, Magnus Therning wrote:
>
>> After some detours I found out that it's actually not a network issue, but
>> rather that the process runs out of filedescriptors. Using =lsof= I can see that
>> it doesn't seem to close /any/ sockets at all, instead they get stuck in a
>> =CLOSE_WAIT= state:
>>
>> #+begin_example
>> COMMAND    PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
>> wd-stats 88674 magnus   23u  IPv4 815196      0t0  TCP ip-192-168-0-9.eu-central-1.compute.internal:60624->52.119.188.213:https (CLOSE_WAIT)
>> wd-stats 88674 magnus   24u  IPv4 811362      0t0  TCP ip-192-168-0-9.eu-central-1.compute.internal:43482->52.119.189.184:https (CLOSE_WAIT)
>> wd-stats 88674 magnus   25u  IPv4 811386      0t0  TCP ip-192-168-0-9.eu-central-1.compute.internal:60628->52.119.188.213:https (CLOSE_WAIT)
>> wd-stats 88674 magnus   26u  IPv4 813527      0t0  TCP ip-192-168-0-9.eu-central-1.compute.internal:43486->52.119.189.184:https (CLOSE_WAIT)
>> ...
>> #+end_example
>
> How many such still open file descriptors did you find?

Hundreds of them.

> (If you run "lsof -n -P -i tcp -a -p $pid", it'll produce
> the output faster, reporting only sockets).
>
> Contrary to other replies, indeed the sockets above are NOT closed in
> your process, otherwise they'd not be associated with a file
> descriptor and would just show up in "netstat", but not "lsof" output.
>
> I don't know what happens inside Amazonka, but typically clients doing
> many concurrent HTTPS calls employ a TlsManager that maintains a
> connection pool, and would avoid opening too many concurrent
> connections, but would also keep a limited number of connections open
> for more requests.

After I reported it to the Amazonka project[1] I found out that it most
likely is a known issue[2].

I have still to confirm that the fix for [2] solves the issue I'm
seeing.

[1]: https://github.com/brendanhay/amazonka/issues/608
[2]: https://github.com/brendanhay/amazonka/issues/490

/M

--
Magnus Therning              OpenPGP: 0x927912051716CE39
email: magnus at therning.org
twitter: magthe              http://magnus.therning.org/

I am always doing that which I cannot do, in order that I may learn
how to do it.
     — Pablo Picasso
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 861 bytes
Desc: not available
URL: <http://mail.haskell.org/pipermail/haskell-cafe/attachments/20201129/28de8ed5/attachment.sig>