V4 Development Tracker

[normj]

V4 Development Tracker

This issue is for tracking the changes being made for V4 of the AWS SDK for .NET. Development of V4 is happening in the v4-development branch in this repository. V4 is an evolutionary major version change with minimal breaking changes so application code using the AWS SDK can upgrade to V4 with low effort.

Update August 15, 2024

First preview packages have been released. Blog post for the release https://aws.amazon.com/blogs/developer/preview-1-of-aws-sdk-for-net-v4/

Breaking changes

The list of breaking changes made in V4.

• .NET Framework 3.5 target has been removed.
• .NET Framework 4.5 target has been updated to 4.7.2
• Properties using value types on classes used for making requests and responses have been changed to use nullable value types. Essentially properties of type bool are changed to bool?.
• Properties using collections on classes used for making requests and responses will now default to null. The V3 behavior of initializing collections can be restored by setting the Amazon.AWSConfigs.InitializeCollections to true. This property also exists in V3 for users that want to try this behavior change before upgrading to V4.
• S3 service clients configured for us-east-1 will no longer be able to access buckets in other regions. Buckets musts be accessed with S3 service clients configured for the region the bucket is in. This change avoids surprises in applications using the S3 service client when under the covers it makes multiple AWS requests to access buckets that are not in service clients configured us-east-1 region.
• The IAM action identifiers have been removed from the Amazon.Auth.AccessControlPolicy.ActionIdentifiers namespace. These identifiers were previously marked as obsolete due to be being out of date with no mechanism to keep them up to date. Code using these ActionIdentifiers should be upgrade to use the string value of the IAM action name.
• The S3 encryption client has been removed from the AWSSDK.S3 package. This client had been marked as obsolete with the encryption client moved to its separate package Amazon.Extensions.S3.Encryption package. The following migration guide to transition to the separate package. https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/s3-encryption-migration.html
• The DetermineServiceURL method on the ClientConfig, the base class of service client configs like AmazonS3Config, was removed. This had been marked obsolete and was tied to an internal endpoint resolution system that was removed in V4. Application code calling this method should switch to using the DetermineServiceOperationEndpoint method on the service client.
• When using credential providers that rely on Amazon Security Token Service (STS) the calls will always use the regional endpoint. V3 by default used us-east-1 regardless of the configured region when running in the public partition. As part of this change the StsRegionalEndpointsValue enum was removed. Any code using that enum should be removed.
• On service client configs the RetryMode defaults to Standard instead of Legacy. The Legacy enum value was removed.
• On service client configs the DefaultConfigurationMode defaults to Standard instead of Legacy. The Legacy enum value was removed.
• DefaultClientConfig in AWSSDK.Extensions.NETCore.Setup no longer extends from service client config base class ClientConfig. The service client config properties have been replicated on DefaultClientConfig using nullable value types to allow detecting when a value has been set on DefaultClientConfig when copying the values to the service client config being created for the service client.
• CloudFront's CookieSigner and UrlSigner have been moved to a separate extension package called AWSSDK.Extensions.CloudFront.Signers to support SSL3 and take a dependency on BouncyCastle.Cryptography.
• EC2's GetDecryptedPassword has been moved to a separate package called AWSSDK.Extensions.EC2.DecryptPassword to support OpenSSL 3 and take a dependency on BouncyCastle.Cryptography.
• The SDK's source copy of BouncyCastle has been removed.
• S3's Tagging-Directive has been exposed as a public property on CopyObjectRequest and is no longer automatically set to COPY. If a user doesn't specify a tagging directive the S3 backend automatically assumes it is COPY but if a user explicitly sets it to null, then we won't set the value at all.
• AmazonS3Config's UseArnRegion will take the value set in the environment variable over that set in the ~/.aws/config file. This follows other config environment variable + config file precedence.

Other noteworthy changes

Removed Code

• The obsolete ECSTaskCredentials class is removed. Please use the GenericContainerCredentials provider which also supports EKS Pod Identities.
• The obsolete StoredProfileAWSCredentials and StoredProfileCredentials is removed. Please use the NetSDKCredentialsFile or the SharedCredentialsFile.
• The obsolete HasCachedAccessTokenAvailable method on the SSOAWSCredentials class is removed.
• The obsolete EnvironmentAWSCredentials is removed. Please use AppConfigAWSCredentials instead.
• The obsolete StoredProfileFederatedCredentials is removed. Please use FederatedAWSCredentials instead.
• The obsolete StoredProfileSAMLCredentials class in SecurityTokenService is removed. Please use FederatedAWSCredentials instead.
• The obsolete EndpointDiscoveryEnabled class is removed. Please use the EnvironmentVariableAWSConfiguration instead.
• The obsolete UseSigV4 property on the AmazonWebServiceRequest class is removed. SignatureVersion will be used directly instead.
• The obsolete TryGetSection methods in ProfileIniFile that don't take in nested properties is removed. Please use the TryGetSection method in ProfileIniFile that accounts for nested properties.
• The obsolete Parameters dictionary in WebServiceRequestEventArgs is obsolete. ParameteCollection will be used instead.
• The obsolete DoesS3BucketExistAsync(string bucketName) in ICoreAmazonS3 is removed because it always uses HTTP. Please use Amazon.S3.Util.AmazonS3Util.DoesS3BucketExistV2Async instead.
• The following methods in AWSSDKUtils are removed ResolveResourcePath, ProtectEncodedSlashUrlEncode, ConvertToUnixEpochMilliSeconds
• EC2_METADATA_SVC, EC2_METADATA_ROOT, EC2_USERDATA_ROOT, EC2_DYNAMICDATA_ROOT, EC2_APITOKEN_URL in EC2InstanceMetadata are removed. The fields were marked as obsolete in v3.
• The obsolete ProfileManager class in Amazon.Util is removed. Please use the SharedCredentialsFile or the NetSDKCredentialsFile instead.
• The following obsolete properties on AWSConfigs are removed: Logging, ResponseLogging, and LogMetrics. Use LoggingConfig instead.
• public static Condition NewCondition(DateComparisonType type, DateTime date) in Amazon.Auth/AccessControlPolicy/ConditionFactory is removed. Please use NewConditionUtc instead.
• In the ClientConfig class, the obsolete DetermineDnsSuffix has been removed. Use the service specific client.DetermineServiceOperationEndpoint method instead. The ReadEntireResponse property has been removed, use the AWSConfigs.LoggingConfig.LogResponses or ClientConfig.LogResponse instead.

DynamoDB Changes

In V4 we're addressing issues around testability, improving configuring the high-level Table and DynamoDBContext clients, and potentially additional fixes that require breaking changes.

• In DynamoDB's high-level programming model, initializing the Table with a mocked IAmazonDynamoDB now returns an InvalidOperationException instead of a NullReferenceException . Async Table methods should now work with a mocked client but you may still see exceptions when calling sync methods from .NET/Core/Standard. See PR #3388, this mostly addresses DynamoDB Table misrepresents dependency on IAmazonDynamoDB #1589.
• In DynamoDB's object persistence programming model, we've separated the shared DynamoDBOperationConfig into new operation-specific objects (SaveConfig, LoadConfig, QueryConfig, etc.). The shared config has grown overtime, and contains properties that don't apply to every operation which can lead to confusion. we've marked the methods that take DynamoDBOperationConfig as obsolete. We won't remove them from V4 yet, though may do so in a future version. See PR #3421
• In DynamoDB's object persistence programming model, we've removed MetadataCachingMode and DisableFetchingTableMetadata from DynamoDBOperationConfig, and did not include these on the new operation-specific configs that were introduced above. These are table-level settings that should be specified on the global AWSConfigsDynamoDB.Context or on DynamoDBContextConfig. See PR #3422
• In DynamoDB's object persistence programming model, DynamoDBOperationConfig no longer inherits from DynamoDBContextConfig. This prevents you from passing a DynamoDBOperationConfig in to the constructor for DynamoDBContext, where some properties on the operation-specific config (such as OverrideTableName) do not apply. See PR #3422
• DynamoDBStreams has been removed from the DynamoDB package and is now in its own package and namespace.
• Community PR #3476 Allow to remove TableNamePrefix on individual operation level. Thanks Oleksandr Liakhevych
• We have added new operation specific interfaces to allow customers to mock DynamoDB operations. See PR #3450. The factory methods on IDynamoDBContext have been updated to return the new interfaces.
  • Object Persistence Programming Model
    • Mock BatchGet operations via IBatchGet and IMultiTableBatchGet interfaces.
    • Mock BatchWrite operations via IBatchWrite and IMultiTableBatchWrite interfaces.
    • Mock TransactGet operations via ITransactGet and IMultiTableTransactGet interfaces.
    • Mock TransactWrite operations via ITransactWrite and IMultiTableTransactWrite interfaces.
    • Mock Scan and Query operations via the IAsyncSearch interface.
  • Document Programming Model
    • Mock Table operations via the ITable interface.
    • Mock Scan and Query operations via the ISearch interface.
    • Mock TransactWrite operations via IDocumentTransactWrite and IMultiTableDocumentTransactWrite interfaces.
    • Mock TransactGet operations via IDocumentTransactGet and IMultiTableDocumentTransactGet interfaces.
    • Mock BatchWrite operations via IDocumentBatchWrite and IMultiTableDocumentBatchWrite interfaces.
    • Mock BatchGet operations via IDocumentBatchGet and IMultiTableDocumentBatchGet interfaces.

---

The following changes were made in the SDK. They do not require changes to applications code using the SDK.

• The following dependencies were added to the .NET Framework 4.7.2 and .NET Standard 2.0 targets
  • System.Buffers
  • System.Memory
  • System.Text.Json
• AWSSDK.Extensions.NETCore.Setup has been made Native AOT compatible.
• In .NET Standard 2.0, .NET Core 3.1 and .NET 8 the AssemblyVersion will match the AssemblyFileVersion. In .NET Framework 4.7.2 target AssemblyVersion will continue to use the V3 pattern of locking the version to the first 2 parts of the the AssemblyFileVersion. This is done because .NET Framework treats the AssemblyVersion as part of the identity assembly. This causes assembly collisions and forced recompilation when mixing versions of the SDK.
• The embedded endpoints.json file in AWSSDK.Core was removed. This was a 1 MB json file that was parsed at startup for the SDK to determine the service endpoint for a region. Endpoint resolution has been replaced with a new system that generates rules into each individual service assembly for determining the service endpoints.
• Community PR #3359 Optimize GetExtension execution time on .NET 8. Thanks Steven Weerdenburg
• Community PR #3293 improving AWSSDK.ToHex performance. Thanks Steven Weerdenburg
• Community PR #3292 improving AWSSDK.CopyTo performance. Thanks Daniel Marbach
• Community PR #3324 optimizing the AWS SigV4 signer. Thanks Daniel Marbach
• Community PR #3363 improving AWSSDKUtils.ParameterAsString performance. Thanks Daniel Marbach
• Community PR #3365 improving AWSSDKUtils.DetermineService performance. Thanks Daniel Marbach
• Community PR #3307 Optimizing AWSSDKUtils.UrlEncode performance. Thanks Daniel Marbach
• Community PR #3425 Avoid allocating byte[] when converting MemoryStream to String. Thanks Paulo Morgado
• Community PR #3423 Refactor user agent construction to avoid creating unnecessary string objects. Thanks Paulo Morgado
• Community PR #3439 Refactor JSON request marshaller to avoid creating unnecessary intermediate strings. Thanks Paulo Morgado
• Community PR #3458 Add ToString methods for better enum string conversion.
  Thanks Paulo Morgado

[Dreamescaper]

Is there any way to test those packages? Some preview nuget source?

[dscpinheiro]

We'll publish the preview packages to NuGet (e.g. <PackageReference Include="AWSSDK.AccessAnalyzer" Version="4.0.0.0-preview" />), we're just working on some changes to our internal build system at the moment (we'll update this issue as soon as the packages are available).

[normj]

First preview packages have been released. Blog post for the release https://aws.amazon.com/blogs/developer/preview-1-of-aws-sdk-for-net-v4/

[martincostello]

Are you able to provide a preview version of Amazon.AspNetCore.DataProtection.SSM that depends on AWSSDK.Extensions.NETCore.Setup?

Trying to enable native AoT for a project that depends on the SSM package gives a bunch of warnings:

ILC : Trim analysis error IL2026: Microsoft.Extensions.DependencyInjection.ExtensionMethods.PersistKeysToAWSSystemsManager(IDataProtectionBuilder,String,Action`1<PersistOptions>): Using member 'Microsoft.Extensions.DependencyInjection.ServiceCollectionExtensions.TryAddAWSService<IAmazonSimpleSystemsManagement>(IServiceCollection,ServiceLifetime)' which has 'RequiresUnreferencedCodeAttribute' can break functionality when trimming application code. The AWSSDK.Extensions.NETCore.Setup package has not been updated to support Native AOT compilations. 
C:\codebuild\tmp\output\src1982900738\src\aws-sdk-net\sdk\src\Core\Amazon.Runtime\ConstantClass.cs(84): Trim analysis error IL2072: Amazon.Runtime.ConstantClass.Intern(): 'type' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields' in call to 'Amazon.Runtime.ConstantClass.LoadFields(Type)'. The return value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to.
  ILC: Method '[AWSSDK.Extensions.NETCore.Setup]Amazon.Extensions.NETCore.Setup.DefaultClientConfig..ctor()' will always throw because: Missing method 'Void Amazon.Runtime.ClientConfig..ctor()'
  ILC: Method '[Amazon.AspNetCore.DataProtection.SSM]Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository+<GetAllElementsAsync>d__9.MoveNext()' will always throw because: Missing method 'Void Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest.set_WithDecryption(Boolean)'

It's possible some of the above are also coming from internal libraries for transient dependencies that haven't been compiled against v4 yet either.

The app is still blocked from 0 AoT warnings due to a dependency on another third-party library that hasn't published some fixes I made yet, but otherwise this SSM package is the final hurdle.

[normj]

@martincostello I don't have a timeline yet but we would like push out preview versions of all of our high level packages like Amazon.AspNetCore.DataProtection.SSM but we have some internal work first we need to do setting up a dual preview pipeline across our repositories.

[martincostello]

FYI this is the error I get from an .NET 8 app running as a Lambda with the latest version of Amazon.AspNetCore.DataProtection.SSM. It builds fine, but then fails at runtime:

System.Security.Cryptography.CryptographicException: An error occurred while trying to encrypt the provided data. Refer to the inner exception for more information.
 ---> System.MissingMethodException: Method not found: 'Void Amazon.SimpleSystemsManagement.Model.GetParametersByPathRequest.set_WithDecryption(Boolean)'.
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElementsAsync()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElementsAsync()
   at System.Threading.Tasks.Task`1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Amazon.AspNetCore.DataProtection.SSM.SSMXmlRepository.GetAllElements()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.GetAllKeys()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   --- End of inner exception stack trace ---
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.Protect(Byte[] plaintext)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgeryTokenSerializer.Serialize(AntiforgeryToken token)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.Serialize(IAntiforgeryFeature antiforgeryFeature)
   at Microsoft.AspNetCore.Antiforgery.DefaultAntiforgery.GetAndStoreTokens(HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.AntiforgeryExtensions.GetHtml(IAntiforgery antiforgery, HttpContext httpContext)
   at Microsoft.AspNetCore.Mvc.TagHelpers.FormTagHelper.Process(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.TagHelpers.TagHelper.ProcessAsync(TagHelperContext context, TagHelperOutput output)
   at Microsoft.AspNetCore.Razor.Runtime.TagHelpers.TagHelperRunner.RunAsync(TagHelperExecutionContext executionContext)
   at AspNetCoreGeneratedDocument.Pages_Shared_SignIn.ExecuteAsync() in /_/src/Codeflowbot/Pages/Shared/SignIn.cshtml:line 13
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageCoreAsync(IRazorPage page, ViewContext context)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderPageAsync(IRazorPage page, ViewContext context, Boolean invokeViewStarts)
   at Microsoft.AspNetCore.Mvc.Razor.RazorView.RenderAsync(ViewContext context)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.ViewFeatures.ViewExecutor.ExecuteAsync(ViewContext viewContext, String contentType, Nullable`1 statusCode)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeResultAsync>g__Logged|22_0(ResourceInvoker invoker, IActionResult result)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at MyApp.MyLoggerMiddleware.Invoke(HttpContext context) in /_/src/Codeflowbot/Middleware/CurrentUserLoggerMiddleware.cs:line 20
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddlewareImpl.<Invoke>g__Awaited|10_0(ExceptionHandlerMiddlewareImpl middleware, HttpContext context, Task task)
   at MyApp.MyMiddleware.Invoke(HttpContext context) in /_/src/Codeflowbot/Middleware/LambdaLoggerMiddleware.cs:line 38
   at Amazon.Lambda.AspNetCoreServer.AbstractAspNetCoreFunction`2.ProcessRequest(ILambdaContext lambdaContext, Object context, InvokeFeatures features, Boolean rethrowUnhandledError)

[normj]

@martincostello A preview version of Amazon.AspNetCore.DataProtection.SSM is out targeting V4 https://www.nuget.org/packages/Amazon.AspNetCore.DataProtection.SSM/4.0.0-preview.1

[martincostello]

Thanks @normj.

I've tested the latest version of the packages with three of our applications collectively using the v4 packages listed below, and I've not encountered any issues:

• Amazon.AspNetCore.DataProtection.SSM
• AWSSDK.DynamoDBv2
• AWSSDK.SimpleSystemsManagement
• AWSSDK.SecurityToken

I've also tried native AoT publishing with two of the three, and there were also no trim warnings.

[normj]

Awesome @martincostello! Thanks for the feedback.

[Dreamescaper]

Would it make sense to drop 'netcoreapp3.1' target framework as well in V4? It is not supported by Microsoft. Anyone who uses it for whatever reason - would still be able to use netstandard2.0 version.