TCP health checks results in proxy.py not responding after sometime

[sshantha22]

The proxy runs fine initially and works as expected. After sometime (even when there is no actual request but just tcp heartbeat requests), the proxy stops working and just prints the following with debug logging enabled:

2024-08-13 00:51:07,106 - pid:2407 [D] acceptor.accept:118 - Accepting new work#23
2024-08-13 00:51:08,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#24
2024-08-13 00:51:17,104 - pid:2407 [D] acceptor.accept:118 - Accepting new work#26
2024-08-13 00:51:18,429 - pid:2407 [D] acceptor.accept:118 - Accepting new work#27
2024-08-13 00:51:20,006 - pid:2407 [D] acceptor.accept:118 - Accepting new work#28
2024-08-13 00:51:27,105 - pid:2407 [D] acceptor.accept:118 - Accepting new work#29
2024-08-13 00:51:28,429 - pid:2407 [D] acceptor.accept:118 - Accepting new work#30
2024-08-13 00:51:37,105 - pid:2407 [D] acceptor.accept:118 - Accepting new work#31
2024-08-13 00:51:38,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#32
2024-08-13 00:51:47,104 - pid:2407 [D] acceptor.accept:118 - Accepting new work#33
2024-08-13 00:51:48,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#34
2024-08-13 00:51:57,105 - pid:2407 [D] acceptor.accept:118 - Accepting new work#35
2024-08-13 00:51:58,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#36
2024-08-13 00:52:07,105 - pid:2407 [D] acceptor.accept:118 - Accepting new work#37
2024-08-13 00:52:08,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#38
2024-08-13 00:52:17,104 - pid:2407 [D] acceptor.accept:118 - Accepting new work#39
2024-08-13 00:52:18,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#40
2024-08-13 00:52:27,105 - pid:2407 [D] acceptor.accept:118 - Accepting new work#41
2024-08-13 00:52:28,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#42
2024-08-13 00:52:37,105 - pid:2407 [D] acceptor.accept:118 - Accepting new work#43
2024-08-13 00:52:38,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#44
2024-08-13 00:52:47,104 - pid:2407 [D] acceptor.accept:118 - Accepting new work#45

Any idea what could be wrong here potentially?

I would expect the following messages atleast:
2024-08-13 00:50:48,430 - pid:2407 [D] acceptor.accept:118 - Accepting new work#22
2024-08-13 00:50:48,443 - pid:2407 [D] tcp_server.init:119 - Work#22 accepted from ::ffff:10.0.182.255:65399
2024-08-13 00:50:48,446 - pid:2407 [D] tcp_server.initialize:135 - Handling connection ::ffff:10.0.182.255:65399

[abhinavsingh]

What exactly do you mean by TCP heartbeat. May be above log is a side-effect of opening up TCP connections but never completing the request?

[sshantha22]

I am running the proxy on an EC2 instance and the target group the instance is part of is doing tcp health check every 5 seconds. The proxy will be running fine for hours together and then will suddenly stop handling connections once accepted. This has happened even when I haven't sent any request to proxy for several hours apart from the healthchecks. It looks like the localfdexecutor exits out of the loop for some weird reason and hence stops handling connections even though the acceptor is still accepting new connections. netstat confimed that the connection is waiting to be closed by the proxy but since localfdexecutor exited, connections are not being closed.

[abhinavsingh]

I think cause if the TCP health check. What might be happening is, your health checks are leaving back open file descriptors. You can tweak the limits using --open-file-limit flag, but any limit will get exhausted by such health checks.

Here is how we deploy health checks for proxy.py based deployments:

1. Write a simple web server plugin which returns 200 OK
2. Configure your health check to hit your HTTP endpoint

I'll recommend you do the same for your setup. Let me know if it helps.

[sshantha22]

Thank you @abhinavsingh. I'll give that a try.