Complete freeze after failure communicating w/ indexer-service

[aasseman]

AA freezes, in this case it stayed frozen for days. A proper crash would be better, so that we know when to restart it.

Giving up subgraph_query_count(...) after 2 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.64.14:7300 ssl:default [Connect call failed ('192.168.64.14', 7300)])
Giving up queries_per_second(...) after 3 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.64.14:7300 ssl:default [Connect call failed ('192.168.64.14', 7300)])
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
    return await self._loop.create_connection(*args, **kwargs)  # type: ignore[return-value]  # noqa
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('192.168.64.14', 7300)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/autoagora/price_multiplier.py", line 111, in price_bandit_loop
    queries_per_second = await environment.queries_per_second(
  File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/autoagora/subgraph_wrapper.py", line 45, in queries_per_second
    query_count_1 = await subgraph_query_count(self.subgraph, metrics_endpoints)
  File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/autoagora/query_metrics.py", line 109, in subgraph_query_count
    async with session.get(endpoint) as response:
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
    raise last_exc
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
  File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
    raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.64.14:7300 ssl:default [Connect call failed ('192.168.64.14', 7300)]