Retry Classifier Customization Guidance

[Velfi]

How the orchestrator models retries

A Retry Strategy is the process by which the orchestrator determines when and how to retry failed requests. Only one retry strategy may be set at any given time. During its operation, the retry strategy relies on a series of Retry Classifiers to determine if and how a failed request should be retried.

Retry classifiers each have a Retry Classifier Priority so that regardless of whether they are set during config or operation construction, they'll always run in a consistent order. Classifiers are each run in turn by the retry strategy. In general, it's better to write retry classifiers that don't conflict with each other and to just use the default priority.

The following example demonstrates how to implement and set an operation-specific retry classifier:

use aws_smithy_runtime_api::client::interceptors::context::InterceptorContext;
use aws_smithy_runtime_api::client::orchestrator::OrchestratorError;
use aws_smithy_runtime_api::client::retries::classifiers::{
    ClassifyRetry, RetryAction, RetryClassifierPriority,
};
use aws_smithy_types::error::metadata::ProvideErrorMetadata;
use aws_smithy_types::retry::ErrorKind;
use std::error::Error as StdError;
use std::marker::PhantomData;
use aws_sdk_s3::config::Config;

const RETRYABLE_ERROR_CODES: &[&str] = [
    // List error codes to be retried here...
];

// When classifying at an operation's error type, classifiers require a generic parameter.
// When classifying the HTTP response alone, no generic is needed.
#[derive(Debug, Default)]
pub struct ErrorCodeClassifier<E> {
    _inner: PhantomData<E>,
}

impl<E> ExampleErrorCodeClassifier<E> {
    pub fn new() -> Self {
        Self {
            _inner: PhantomData,
        }
    }
}

impl<E> ClassifyRetry for ExampleErrorCodeClassifier<E>
where
    // Adding a trait bound for ProvideErrorMetadata allows us to inspect the error code.
    E: StdError + ProvideErrorMetadata + Send + Sync + 'static,
{
    fn classify_retry(&self, ctx: &InterceptorContext) -> RetryAction {
        // Check for a result
        let output_or_error = ctx.output_or_error();
        // Check for an error
        let error = match output_or_error {
            Some(Ok(_)) | None => return RetryAction::NoActionIndicated,
              Some(Err(err)) => err,
        };

        // Downcast the generic error and extract the code
        let error_code = OrchestratorError::as_operation_error(error)
            .and_then(|err| err.downcast_ref::<E>())
            .and_then(|err| err.code());

        // If this error's code is in our list, return an action that tells the RetryStrategy to retry this request.
        if let Some(error_code) = error_code {
            if RETRYABLE_ERROR_CODES.contains(&error_code) {
                return RetryAction::transient_error();
            }
        }

        // Otherwise, return that no action is indicated i.e. that this classifier doesn't require a retry.
        // Another classifier may still classify this response as retryable.
        RetryAction::NoActionIndicated
    }
    
    fn name(&self) -> &'static str { "Example Error Code Classifier" }

    // The priority trait method is left unimplemented here, so this classifier will have the default priority.
}

let sdk_config = aws_config::load_from_env().await;
let client = aws_sdk_s3::Client::new(&sdk_config);
let some_object = client
    .get_object()
    .bucket("bucket")
    .key("key")
    .customize()
    .config_override(aws_sdk_s3::config::Config::builder().retry_classifier(ExampleErrorCodeClassifier::<GetObjectError>::new()))
    .send()
    .await

Generally speaking, classifiers that require knowledge of a specific operation error type should set at the operation level. Otherwise, they'll be run against errors that they can never classify, wasting effort. Classifiers that don't require specific types or that only act on the HTTP response should be set at the service config level.

Default Classifiers

Smithy clients have three classifiers enabled by default:

• ModeledAsRetryableClassifier: Checks for errors that are marked as retryable in the smithy model. If one is encountered, returns Some(RetryAction::Retry(ErrorKind)). Otherwise, returns None. Requires a parsed response.
• TransientErrorClassifier: Checks for timeout, IO, and connector errors. If one is encountered, returns Some(RetryAction::Retry(ErrorKind::TransientError)). Otherwise, returns None. Requires a parsed response.
• HttpStatusCodeClassifier: Checks the HTTP response's status code. By default this classifies 500, 502, 503, and 504 errors as Some(RetryAction::Retry(ErrorKind::TransientError)). Otherwise, returns None. The list of retryable status codes may be customized when creating this classifier with the HttpStatusCodeClassifier::new_from_codes method.

AWS clients enable the three smithy classifiers as well as one more by default:

• AwsErrorCodeClassifier: Checks for errors with AWS error codes marking them as either transient or throttling errors. If one is encountered, returns Some(RetryAction::Retry(ErrorKind)). Otherwise, returns None. Requires a parsed response. This classifier will also check the HTTP response for an x-amz-retry-after header. If one is set, the retry will be attempted after a the specified delay.

The priority order of these classifiers is as follows:

1. (highest priority) TransientErrorClassifier
2. ModeledAsRetryableClassifier
3. AwsErrorCodeClassifier
4. (lowest priority) HttpStatusCodeClassifier