Adding a configuration for max retries to handle system wide outages

[andrewgaun]

• Algolia Client Version: 3.16.6
• Language Version: Java

Description

Correct me if I am mistaken but HttpTransport.executeWithRetry will before sending a request, increase the timeout for the request based on the number of retries. After the response is received, it uses RetryStrategy to determine what to do with the results.

The function RetryStrategy.decide will in the case of a request time out, will always retry.

During an Indexer service outage in July 13th, it appears that was happening for the entire length of the outage causing the Promises to hang and never fail. This becomes a pretty big issue when using BatchIndexingResponse.waitTask() like in the API examples.

Proposed change

1. Adding a new configurable property to ConfigBase called something along the lines of maxRetriesPerHost and update RetryStrategy to save that value in its constructor.
2. Update RetryStrategy.decide to something along the lines of

} else if (response.isTimedOut()) {
    boolean shouldKeepUp = maxRetriesPerHost == null  || tryableHost.getRetryCount() <= maxRetriesPerHost;
    tryableHost.setUp(shouldKeepUp);
    tryableHost.setLastUse(AlgoliaUtils.nowUTC());
    tryableHost.incrementRetryCount();
    return RetryOutcome.RETRY;
}

Why make the change

During the time of the indexing outage, it was not clear immediately why our requests were stuck. Looking at the code it seems like if the servers actually responded with server errors, the StatefulHosts would have been turned off one by one which would be closer to what I would expect from an outage. The current logic has no way of determining this type of issue, causing ever expanding timeouts without any way of intervening.