RETRY implements a backoff strategy that differs from Google recommended strategy - customise retry strategy?

[MarkEdmondson1234]

This issue has been raised on googleAuthR: MarkEdmondson1234/googleAuthR#186

Is it possible to have some more customisation on how the retry strategy occurs? At the moment it means that too many requests happen for Google APIs in particular, and perhaps other APIs may also have differing backoff rules.

In the Google API case, the stipulation is: "2^attempt + random(0, pause_base)" whereas httr's RETRY uses "randomly wait between 0 and pause_base * 2 ^ attempt seconds"

The difference means that by the 3rd retry the difference is minimum 8s vs between 0-8seconds which means the 3rd retry is always too soon.

[jennybc]

As a point of reference, gargle now has a branch where I implement a Google-ish retry strategy:

r-lib/gargle@master...retry

[jennybc]

And to address one of your points more specifically: I'm not sure there's really a "Google party line" on how exponential backoff should be implemented. They are pretty clear that they want exponential backoff, morally speaking.

In the Google API case, the stipulation is: "2^attempt + random(0, pause_base)" whereas httr's RETRY uses "randomly wait between 0 and pause_base * 2 ^ attempt seconds"

I dove into all these links in the @seealso section of gargle::request_retry() in the branch referred to above:

• https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
• https://tech.channable.com/posts/2020-02-05-opnieuw.html
• https://github.com/channable/opnieuw
• https://cloud.google.com/storage/docs/exponential-backoff
• https://cloud.google.com/storage/docs/gsutil/addlhelp/RetryHandlingStrategy
• https://tools.ietf.org/html/rfc7231#section-7.1.3
• https://developers.google.com/sheets/api/limits
• https://googleapis.dev/python/google-api-core/latest/retry.html

My memory is a bit foggy, but I'm pretty sure that I saw examples of a exponentially increasing deterministic wait + random jitter AND random wait with an exponentially increasing support, even within the Google links.

And then, in my case, there was a very special important case for Google Sheets (quota violation), where I have to dig the wait time out of a message 😬

[MarkEdmondson1234]

Wow lots of thought on it already, great to see. I did wonder if this would be Google specific so look at it in gargle but it seems httr would be a nice place to have a more generic solution.

Perhaps being able to pass a numeric sequence or a function that makes a sequence to RETRY would be most general way to support all future possibilities, the sequence being the number of seconds between each try.

[jennybc]

I think any substantial innovation / upgrading might be a task for httr2 (https://github.com/r-lib/httr2/issues), which is mostly just an issue target at this point. I definitely thought of my gargle retry work as 50% for gargle-using packages today and 50% research for more flexible retry behaviour in httr or its successor.

[hadley]

httr has been superseded in favour of httr2, so is no longer under active development. If this problem is still important to you in httr2, I'd suggest filing an issue offer there 😄 — but req_retry() implements "truncated exponential backoff with full jitter" and you can provide your own strategy if that doesn't work for you. Thanks for using httr!