socket.timeout: timed out

[huwaizatahir2]

Bug Description
Recently I have upgraded my Flask-SocketIO package from 4.3.2 to 5.3.4 and I am starting to get frequent timeout errors.
Further related upgraded packages.
greenlet==1.0.0 => greenlet==2.0.2
eventlet==0.32.0 => eventlet==0.33.3

Logs

[2024-09-18 08:34:43 +0000] [3201252] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 64, in handle
    util.reraise(*sys.exc_info())
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/gunicorn/util.py", line 640, in reraise
    raise value
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 54, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 129, in handle_request
    util.reraise(*sys.exc_info())
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/gunicorn/util.py", line 640, in reraise
    raise value
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/gunicorn/workers/base_async.py", line 107, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/flask/app.py", line 2213, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/flask_socketio/__init__.py", line 43, in __call__
    return super(_SocketIOMiddleware, self).__call__(environ,
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/middleware.py", line 63, in __call__
    return self.engineio_app.handle_request(environ, start_response)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/socketio/server.py", line 607, in handle_request
    return self.eio.handle_request(environ, start_response)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/server.py", line 410, in handle_request
    packets = socket.handle_get_request(
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/socket.py", line 103, in handle_get_request
    return getattr(self, '_upgrade_' + transport)(environ,
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/socket.py", line 158, in _upgrade_websocket
    return ws(environ, start_response)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/async_drivers/eventlet.py", line 20, in __call__
    return super(WebSocketWSGI, self).__call__(environ, start_response)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/websocket.py", line 136, in __call__
    self.handler(ws)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/socket.py", line 179, in _websocket_handler
    pkt = websocket_wait()
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/engineio/socket.py", line 163, in websocket_wait
    data = ws.wait()
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/websocket.py", line 820, in wait
    for i in self.iterator:
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/websocket.py", line 661, in _iter_frames
    message = self._recv_frame(message=fragmented_message)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/websocket.py", line 687, in _recv_frame
    header = recv(2)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/websocket.py", line 589, in _get_bytes
    d = self.socket.recv(numbytes - len(data))
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/greenio/base.py", line 352, in recv
    return self._recv_loop(self.fd.recv, b'', bufsize, flags)
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/greenio/base.py", line 346, in _recv_loop
    self._read_trampoline()
File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/greenio/base.py", line 314, in _read_trampoline
    self._trampoline(
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/greenio/base.py", line 206, in _trampoline
    return trampoline(fd, read=read, write=write, timeout=timeout,
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/hubs/__init__.py", line 155, in trampoline
    return hub.switch()
  File "/home/xiangqi/xiangqi_venv/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
socket.timeout: timed out

Configurations on code level

✅ Monkey Patching on manage.py eventlet.monkey_patch()
✅ Initialization => `socketio = SocketIO()

Supervisor configuration
As mentioned in Flask Socket IO deployment documentation, I used Gunicorn web server for the deployment. As mentioned it is not possible to use more than one worker process when using this web server So i instead I created 3 single-worker instances to use multiple worker processes with gunicorn on different ports.

/home/xiangqi/xiangqi_venv/bin/gunicorn -b 0.0.0.0:5001 --worker-connections 25000 --worker-class eventlet -w 1 manage:app
/home/xiangqi/xiangqi_venv/bin/gunicorn -b 0.0.0.0:5002 --worker-connections 25000 --worker-class eventlet -w 1 manage:app
/home/xiangqi/xiangqi_venv/bin/gunicorn -b 0.0.0.0:5003 --worker-connections 25000 --worker-class eventlet -w 1 manage:app

Additional context
I am thinking of doing 2 things, please do help out and let me know if the direction is right?

1. Adding socketio.sleep(0) on handlers I think are time consuming as mentioned by @miguelgrinberg here, although i am doubtul on this since my average response time for maximum time consuming handler is 25.32 ms
2. Default Ping timeout is 5 seconds, I can set ping_timeout to 20 seconds for that matter.

[miguelgrinberg]

I would suggest you migrate off of eventlet, which is not being well maintained. The threading async mode is the most likely to play nice with your CPU demanding tasks. You can also try gevent if you prefer to continue using greenlets.

[huwaizatahir2]

Thanks for the suggestion, indeed the support for new features is completely eradicated in eventlet. I do have 3 follow up questions on your answer though and i would appreciate your feedback.

1. Lets say If i decide to go with threading, is there any general rule of thumb for number of threads gunicorn -w 1 --threads 100 module:app i.e. lets say I have Socket IO RPM of about 1 - 3 K
2. Do we need to monkey patch gevent on code level as well just like we do with eventlet?
3. Do you think adding socketio.sleep(0) or increasing the ping timeout would not help in this case.

[miguelgrinberg]

1. The maximum number of threads is determined by the RAM that you have and the needs of your application. This is the same as with greenlets under eventlet/gevent. You were using 25K which seems unrealistically high.
2. gevent also provides monkey patching. Wether you need it or not depends on your application, same as with eventlet.
3. The ping timeout has nothing to do with anything and should normally be left alone. A sleep(0) helps to achieve concurrency in CPU heavy code when using gevent or eventlet. Sleeping under threading is not necessary.

[huwaizatahir2]

@miguelgrinberg thanks for the insights 👍