Deleting S3 object fails on .NET for iOS. #3140

joduss · 2024-01-05T13:39:38Z

Describe the bug

We are using the AWS SDK for .Net in an iOS application built using .NET 8.

We are observing a possible bug when trying to delete an object in a bucket:

In the iOS application, we constantly get an exception (AmazonS3Exception: A header you provided implies functionality that is not implemented) while the same code works fine in a .NET 8 console application. Before migrating the app from Xamarin.iOS to .NET 8, it used to work fine as well.

Expected Behavior

The object should be successfully deleted.

Current Behavior

The following exception is thrown:

AmazonS3Exception: A header you provided implies functionality that is not implemented
                     ▷ Trace:
                       ▷ at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
                       ▷ at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
                       ▷ at Amazon.Runtime.Internal.ExceptionHandler`1.<HandleAsync>d__6[[Amazon.Runtime.Internal.HttpErrorResponseException, AWSSDK.Core, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
                       ▷ at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.Signer.<InvokeAsync>d__1`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                       ▷ at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                     ► HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
                       ▷ Trace:
                         ▷ at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
                         ▷ at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
                         ▷ at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1[[System.Net.Http.HttpContent, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a],[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.Runtime.Internal.RedirectHandler.<InvokeAsync>d__1`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.Runtime.Internal.Unmarshaller.<InvokeAsync>d__3`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.S3.Internal.AmazonS3ResponseHandler.<InvokeAsync>d__1`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
                         ▷ at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()

Reproduction Steps

Use the following code in an application, replacing uppercased values.

var bucket = "BUCKET_NAME";
var credentials = new BasicAWSCredentials("ACCESS_KEY", "SECRET_KEY");

using var s3Client = new AmazonS3Client(credentials, BUCKET_REGION);

var deleteObjectRequest = new DeleteObjectRequest {
    BucketName = bucket,
    Key = "OBJECT_ID",
};

try {
    await s3Client.DeleteObjectAsync(deleteObjectRequest);
} catch (Exception e) {
    Console.WriteLine(e.StackTrace);
}

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.305.5

Targeted .NET Platform

.NET8-iOS

Operating System and version

iOS 17

The text was updated successfully, but these errors were encountered:

ashishdhingra · 2024-01-05T15:17:35Z

@joduss Good morning. The mentioned error is returned by the S3 service, refer S3: List of error codes. Perhaps, somehow on iOS, there might be erroneous header being sent to S3 service which is causing the error.

Is it Xamarin iOS application targeting .NET 8?
Are you using the same object key in iOS application as in working Console application?
Does object key contain any special characters (like double quote)?
Could you please enable verbose logging using the code below and share the detailed logs which might capture some information sent down to the wire:

Amazon.AWSConfigs.LoggingConfig.LogResponses = Amazon.ResponseLoggingOption.Always;
Amazon.AWSConfigs.LoggingConfig.LogTo = Amazon.LoggingOptions.Console;
Amazon.AWSConfigs.AddTraceListener("Amazon", new System.Diagnostics.ConsoleTraceListener());

Thanks,
Ashish

joduss · 2024-01-05T16:14:58Z

Now regarding your questions:

Is it Xamarin iOS application targeting .NET 8?
- Yes it is.
Are you using the same object key in iOS application as in working Console application?
- Yes

I will share the logs next week.

Workaround (Does not work)

I thought I found a workaround. Somehow it did work for a while but does not anymore:

Initial answer: I did found in the meantime a workaround by instantiating the client in a different way and specifying the DefaultConfigurationMode

new AmazonS3Client(
   Credential.AccessKey,
   Credential.SecretKey,
   new AmazonS3Config {
      RegionEndpoint = RegionEndpoint.GetBySystemName(regionResponse.Location),
      DefaultConfigurationMode = DefaultConfigurationMode.Standard // This fixes things.
   }
);

joduss · 2024-01-08T08:51:16Z

Here are the logs:

verbose.log

ashishdhingra · 2024-01-08T14:23:06Z

@joduss Good morning. Thanks for the response and confirming that the workaround initially shared by you in #3140 (comment) when setting DefaultConfigurationMode to DefaultConfigurationMode.Standard, since we also thought that the error and setting configuration mode explicitly were unrelated.

In the logs, I see the below error:

...
2024-01-08 09:48:07.676911+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.Internal.ErrorHandler.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
2024-01-08 09:48:07.685057+0100 My-App.iOS[57738:6780336] AmazonS3Client 29|2024-01-08T09:48:07.684Z|ERROR|Received error response: [<?xml version="1.0" encoding="UTF-8"?>
2024-01-08 09:48:07.685280+0100 My-App.iOS[57738:6780336] <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>If-Modified-Since</Header><RequestId>BH3YCB93BZMKV75S</RequestId><HostId>OfT15rgmuSK9S8dt+HlGuRFrki/e9NOmX5sAZOvGf1olr5SyKPv9zmOPGI4DaoGMjRUaaCDnd0cBo3IyyiIS+A==</HostId></Error>] --> Amazon.S3.AmazonS3Exception: A header you provided implies functionality that is not implemented
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
2024-01-08 09:48:07.685448+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
...
...
   --- End of inner exception stack trace ---
2024-01-08 09:48:07.696940+0100 My-App.iOS[57738:6780336] AmazonS3Client 30|2024-01-08T09:48:07.695Z|ERROR|AmazonS3Exception making request DeleteObjectRequest to https://dev-bucket.s3.eu-central-1.amazonaws.com/. Attempt 1. --> Amazon.S3.AmazonS3Exception: A header you provided implies functionality that is not implemented
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
2024-01-08 09:48:07.697156+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
2024-01-08 09:48:07.697360+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.Internal.HttpHandler`1.<InvokeAsync>d__9`1[[System.Net.Http.HttpContent, System.Net.Http, Version=8.0.0.0, Culture=neutral, 
...
...
2024-01-08 09:48:07.699023+0100 My-App.iOS[57738:6780336]    at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.<InvokeAsync>d__5`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
2024-01-08 09:48:07.699157+0100 My-App.iOS[57738:6780336]    at Amazon.Runtime.Internal.EndpointDiscoveryHandler.<InvokeAsync>d__2`1[[Amazon.S3.Model.DeleteObjectResponse, AWSSDK.S3, Version=3.3.0.0, Culture=neutral, PublicKeyToken=885c28607f98e604]].MoveNext()
...

It mentions error <Error><Code>NotImplemented</Code><Message>A header you provided implies functionality that is not implemented</Message><Header>If-Modified-Since</Header><RequestId>BH3YCB93BZMKV75S</RequestId><HostId>OfT15rgmuSK9S8dt+HlGuRFrki/e9NOmX5sAZOvGf1olr5SyKPv9zmOPGI4DaoGMjRUaaCDnd0cBo3IyyiIS+A==</HostId></Error>

I'm unsure if If-Modified-Since is causing the issue and how it is being populated for your scenario.

Could you please share the following:

Are you using the same bucket for Console and Xamarin iOS app scenario?
Is your bucket S3 Express bucket? Does it has directory style bucket? (refer Directory buckets section in What is S3 Express One Zone?)
What does your object key looks like? (in other words, does it contain any special characters?)

Similar issue aws/aws-sdk-js#2787, but with no concrete information about the root cause or resolution.

Thanks,
Ashish

joduss · 2024-01-08T14:35:42Z

Are you using the same bucket for Console and Xamarin iOS app scenario?
- I am using the same bucket and trying to delete the same object.
- The object was uploaded sometimes using the AWS SDK for .NET, sometimes using Cyberduck.
Is your bucket S3 Express bucket? Does it has directory style bucket? (refer Directory buckets section in What is S3 Express One Zone?)
- No
What does your object key looks like? (in other words, does it contain any special characters?)
- The object is actually a zip file with the extension being ".appname-instruction". The id of that object is "devices/gQFYdfd8vEil8xyDYpV7GQ/iHJ4KEAlmkWktafe84V2WA.appname-instruction". I have replace the actual application name by "appname". The real name is using only chars in [a-z]. So no special characters in the identifier.

muhammad-othman · 2024-01-10T16:22:35Z

Good day @joduss,
Based on the error that you are getting, I suspect that something is modifying the request headers before it is sent to S3, maybe this related to iOS or something else.
Can you intercept the request and provide it including the headers?

joduss · 2024-01-11T08:07:06Z

This is the request I captured from the .Net iOS app:

DELETE https://bucket-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/iHJ4KEAlmkWktafe84V2WA.myapp-instruction HTTP/1.1
Host: bucket-name.s3.eu-central-1.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/20240105/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=d46489c822f6079ca995f8a22d49c2a0b7d43dab09979ea735540bf547184aa0
If-Modified-Since: Fri, 05 Jan 2024 13:21:57 GMT
X-Amz-Date: 20240105T153804Z
Accept-Encoding: gzip, deflate, br
amz-sdk-invocation-id: 7acaa70f-ec17-402b-942e-8e94f9f7f75c
amz-sdk-request: attempt=1; max=5
Accept-Language: en-US,en;q=0.9
If-None-Match: "13f6daf619d5ae220656c30c7d63c21c"
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
User-Agent: aws-sdk-dotnet-coreclr/3.7.305.5 ua/2.0 os/other md/Darwin 22.6.0 Darwin Kernel Version 22.6.0- Wed Oct 4 21-25-26 PDT 2023- root-xnu-8796.141.3.701.17~4/RELEASE_X86_64#17.0.1 md/ARCH#X64 lang/.NET_Core#8.0.0 md/aws-sdk-dotnet-core#3.7.301.0 api/S3#3.7.305.5 ft/aot cfg/retry-mode#legacy md/ClientAsync
Content-Length: 0
Connection: keep-alive

Now for comparison, the request made the console app:

DELETE https://bucket-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/iHJ4KEAlmkWktafe84V2WA.myapp-instruction HTTP/1.1
User-Agent: aws-sdk-dotnet-coreclr/3.7.305.5 ua/2.0 os/macos#13.6.1 md/ARCH#X64 lang/.NET_Core#8.0.0 md/aws-sdk-dotnet-core#3.7.301.0 api/S3#3.7.305.5 cfg/retry-mode#legacy md/ClientAsync
amz-sdk-invocation-id: d1ea090b-25a3-46de-ab7d-39c9b87aa6f9
amz-sdk-request: attempt=1; max=5
Host: bucket-name.s3.eu-central-1.amazonaws.com
X-Amz-Date: 20240105T153814Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/20240105/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=f01e8c1e7ccb49f8948dfdea33120aaa3438cf8c9f7d3ddd60a1fd00681197e5

joduss · 2024-01-11T10:57:15Z

There is one additional observation I made:

If the file is very small, the delete requests fails as explained above. If the file is larger, it succeeds. The header differs depending if the request is to delete a large file or a small file.

Header of request to delete a file of size 1085 bytes:

DELETE https://backup-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/a.myapp-instruction HTTP/1.1
Host: backup-name.s3.eu-central-1.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_LEY/20240111/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=a2cda8ee204b09aaba7fa6ec890f3c4474aaeb77b61418e1c7975d03ce9b26ba
If-Modified-Since: Thu, 11 Jan 2024 10:42:05 GMT
X-Amz-Date: 20240111T104209Z
Accept-Encoding: gzip, deflate, br
amz-sdk-invocation-id: 188175ed-6834-4c9a-b103-159915b80472
amz-sdk-request: attempt=1; max=5
Accept-Language: en-US,en;q=0.9
If-None-Match: "9aeccf7335aa2afe20282060ae026371"
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
User-Agent: aws-sdk-dotnet-coreclr/3.7.304.0 aws-sdk-dotnet-core/3.7.300.11 .NET_Core/8.0.1 OS/Darwin_22.6.0_Darwin_Kernel_Version_22.6.0:_Wed_Oct__4_21:25:26_PDT_2023;_root:xnu-8796.141.3.701.17~4/RELEASE_X86_64 ft/aot ClientAsync
Content-Length: 0
Connection: keep-alive

Header of request to delete a file of size 903'440 bytes:

DELETE https://backup-name.s3.eu-central-1.amazonaws.com/devices/gQFYdfd8vEil8xyDYpV7GQ/b.myapp-instruction HTTP/1.1
Host: backup-name.s3.eu-central-1.amazonaws.com
Accept: */*
Authorization: AWS4-HMAC-SHA256 Credential=ACCESS_KEY/20240111/eu-central-1/s3/aws4_request, SignedHeaders=host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9bfe86c01072568dfda3b2154a5af084af0b4ea77c53aec9820bcfe4bf42acd0
X-Amz-Date: 20240111T105236Z
Accept-Encoding: gzip, deflate, br
amz-sdk-invocation-id: 48057b06-5d22-4157-b4a4-adbee04b0b1c
amz-sdk-request: attempt=1; max=5
Accept-Language: en-US,en;q=0.9
Content-Length: 0
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
User-Agent: aws-sdk-dotnet-coreclr/3.7.304.0 aws-sdk-dotnet-core/3.7.300.11 .NET_Core/8.0.1 OS/Darwin_22.6.0_Darwin_Kernel_Version_22.6.0:_Wed_Oct__4_21:25:26_PDT_2023;_root:xnu-8796.141.3.701.17~4/RELEASE_X86_64 ft/aot ClientAsync
Connection: keep-alive

muhammad-othman · 2024-01-11T17:29:42Z

Thank you for sharing the requests, I couldn't find anything that might cause this issue from the SDK side.
I've reached out to S3 with your request to figure out the root cause, will let you know when then respond.

ashishdhingra · 2024-03-15T17:07:11Z

V1197046386

joduss added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 5, 2024

ashishdhingra added the s3 label Jan 5, 2024

ashishdhingra added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Jan 5, 2024

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jan 6, 2024

ashishdhingra added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jan 8, 2024

github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jan 9, 2024

ashishdhingra added needs-reproduction This issue needs reproduction. needs-investigation labels Jan 10, 2024

ashishdhingra added the needs-review label Mar 8, 2024

ashishdhingra added service-api This issue is due to a problem in a service API, not the SDK implementation. and removed needs-review labels Mar 15, 2024

ashishdhingra added the p2 This is a standard priority issue label Jun 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deleting S3 object fails on .NET for iOS. #3140

Deleting S3 object fails on .NET for iOS. #3140

joduss commented Jan 5, 2024 •

edited

Loading

ashishdhingra commented Jan 5, 2024 •

edited

Loading

joduss commented Jan 5, 2024 •

edited

Loading

joduss commented Jan 8, 2024

ashishdhingra commented Jan 8, 2024 •

edited

Loading

joduss commented Jan 8, 2024

muhammad-othman commented Jan 10, 2024

joduss commented Jan 11, 2024 •

edited

Loading

joduss commented Jan 11, 2024

muhammad-othman commented Jan 11, 2024

ashishdhingra commented Mar 15, 2024

Deleting S3 object fails on .NET for iOS. #3140

Deleting S3 object fails on .NET for iOS. #3140

Comments

joduss commented Jan 5, 2024 • edited Loading

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS .NET SDK and/or Package version used

Targeted .NET Platform

Operating System and version

ashishdhingra commented Jan 5, 2024 • edited Loading

joduss commented Jan 5, 2024 • edited Loading

joduss commented Jan 8, 2024

ashishdhingra commented Jan 8, 2024 • edited Loading

joduss commented Jan 8, 2024

muhammad-othman commented Jan 10, 2024

joduss commented Jan 11, 2024 • edited Loading

joduss commented Jan 11, 2024

muhammad-othman commented Jan 11, 2024

ashishdhingra commented Mar 15, 2024

joduss commented Jan 5, 2024 •

edited

Loading

ashishdhingra commented Jan 5, 2024 •

edited

Loading

joduss commented Jan 5, 2024 •

edited

Loading

ashishdhingra commented Jan 8, 2024 •

edited

Loading

joduss commented Jan 11, 2024 •

edited

Loading