Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Content-SHA256 hash missmatch with crtBuilder() in uploadFile Request #5650

Open
1 task
chetanyachopra22 opened this issue Oct 9, 2024 · 3 comments
Open
1 task

Content-SHA256 hash missmatch with crtBuilder() in uploadFile Request #5650

chetanyachopra22 opened this issue Oct 9, 2024 · 3 comments
Assignees
@debora-ito
Labels
bug This issue is a bug. crt-client p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days.

Comments

@chetanyachopra22
Copy link

Describe the bug

I am seeing hash missmatch exception while using crtBuilder() but am able to upload file successfully using just builder() method i believe that is NettyNioAsyncHttpClient

   public S3AsyncClient amazonS3Client() throws URISyntaxException, KeyManagementException, NoSuchAlgorithmException {
      AwsBasicCredentials credentials = AwsBasicCredentials.create(accessKey, secretKey);

        // Create a trust manager that does not validate certificate chains
        TrustManager[] trustAllCerts = new TrustManager[]{
                new X509TrustManager() {
                    public java.security.cert.X509Certificate[] getAcceptedIssuers() {
                        return null;
                    }
                    public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType) {}
                    public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType) {}
                }
        };

        // Install the all-trusting trust manager
        SSLContext sc = SSLContext.getInstance("SSL");
        sc.init(null, trustAllCerts, new java.security.SecureRandom());

        // Set the default SSL socket factory to the all-trusting SSL context
        HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
        return S3AsyncClient
                .crtBuilder()
                .credentialsProvider(StaticCredentialsProvider.create(credentials))
                .region(Region.EU_WEST_1)
                .endpointOverride(new URI(endpoint))
                .forcePathStyle(true)
                .build();
      }
      
 public void uploadMultiPart(int size) {
      final String fileName;
      String fileAbsolutePath = fetchFilePath(size)[1];
      PutObjectRequest putObjectRequest = PutObjectRequest.builder()
              .bucket(awsS3Configuration.bucketName)
              .key("inbound/test.txt")
              .build();

      try(S3TransferManager transferManager = createTransferManager()) {
         UploadFileRequest uploadFileRequest = UploadFileRequest.builder()
                 .putObjectRequest(putObjectRequest)
                  .source(Paths.get(fileAbsolutePath))
                 .build();
         try {
            transferManager.uploadFile(uploadFileRequest).completionFuture().join();
            System.out.println("Upload completed!");
         } catch (Exception e) {
            e.printStackTrace();
         }
      } catch (Exception e) {
         log.error("Error while listing files via Storage Provider {}", e.getMessage(), e);
         throw new RuntimeException("Error While downloading Objects from Shared Storage", e);
      }
   }

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

It should return 200 and upload file to target

Current Behavior

I see this error for above code

ig.env_IS_UNDEFINED 2024-10-09 15:10:53,434 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@22eed5ac, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@5e3cf794, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@39e33f0a, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@6a6acd5b, software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor@3efd260f, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@744f613, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e, software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@4cd91316, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@6c0d80f0, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@37bd949d, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@3060b33e, software.amazon.awssdk.services.s3.internal.handlers.S3ExpressChecksumInterceptor@34a55787, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@34af4be3, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@6fed3897, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5a219f17, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@44a6f2bb, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@a98599f, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@c37e1db, software.amazon.awssdk.services.s3.internal.handlers.ObjectMetadataInterceptor@5a381e6d, software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$ValidateRequestInterceptor@6164fcaa, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$AttachHttpAttributesExecutionInterceptor@52eee064]
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,434 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Creating an interceptor chain that will apply interceptors in the following order: [software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@22eed5ac, software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@5e3cf794, software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@39e33f0a, software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@6a6acd5b, software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor@3efd260f, software.amazon.awssdk.services.s3.endpoints.internal.S3ResolveEndpointInterceptor@744f613, software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e, software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f, software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@4cd91316, software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@6c0d80f0, software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@37bd949d, software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@3060b33e, software.amazon.awssdk.services.s3.internal.handlers.S3ExpressChecksumInterceptor@34a55787, software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@34af4be3, software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@6fed3897, software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@5a219f17, software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@44a6f2bb, software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@a98599f, software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@c37e1db, software.amazon.awssdk.services.s3.internal.handlers.ObjectMetadataInterceptor@5a381e6d, software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$ValidateRequestInterceptor@6164fcaa, software.amazon.awssdk.services.s3.internal.crt.DefaultS3CrtAsyncClient$AttachHttpAttributesExecutionInterceptor@52eee064]
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,447 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb' modified the message with its modifyRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,447 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@29781cbb' modified the message with its modifyRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,463 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,463 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.endpoints.internal.S3RequestSetEndpointInterceptor@277f829e' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,464 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,464 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain - Interceptor 'software.amazon.awssdk.services.s3.internal.handlers.StreamingRequestInterceptor@1bc3933f' modified the message with its modifyHttpRequest method.
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,484 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=instrument-management.s3.tst.inf.inaut.io, port=9021, encodedPath=/instrument-reporting-service/inbound/test.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
ig.env_IS_UNDEFINED 2024-10-09 15:10:53,484 [http-nio-8080-exec-1] DEBUG software.amazon.awssdk.request - Sending Request: DefaultSdkHttpFullRequest(httpMethod=PUT, protocol=https, host=instrument-management.s3.tst.inf.inaut.io, port=9021, encodedPath=/instrument-reporting-service/inbound/test.txt, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, Expect, User-Agent], queryParameters=[])
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.requestId - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.requestId - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.request - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
ig.env_IS_UNDEFINED 2024-10-09 15:10:55,184 [AwsEventLoop 16] DEBUG software.amazon.awssdk.request - Received failed response: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b
java.util.concurrent.CompletionException: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b)
	at software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:65)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:103)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:184)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:170)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:110)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:253)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The Content-SHA256 you specified did not match what we received (Service: S3, Status Code: 400, Request ID: 0a043343:1921e5187b2:2052:1, Extended Request ID: c98bd6f98a178bc19bb33081cfc42025bc4260f72f182ac0bc24b8bb81bc120b)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
	at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
	at software.amazon.awssdk.utils.async.SimplePublisher.doProcessQueue(SimplePublisher.java:275)
	at software.amazon.awssdk.utils.async.SimplePublisher.processEventQueue(SimplePublisher.java:224)
	at software.amazon.awssdk.utils.async.SimplePublisher.complete(SimplePublisher.java:157)
	at java.base/java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:819)
	at java.base/java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:803)
	at java.base/java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2195)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onErrorResponseComplete(S3CrtResponseHandlerAdapter.java:181)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:160)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:129)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:25)
ig.env_IS_UNDEFINED 2024-10-09 15:12:07,750 [Thread-8] INFO  org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor - Shutting down ExecutorService 'applicationTaskExecutor'

Reproduction Steps

just use code above with defined dependencies

Possible Solution

No response

Additional Information/Context

	<dependencyManagement>
		<dependencies>

			<dependency>
				<groupId>software.amazon.awssdk</groupId>
				<artifactId>bom</artifactId>
				<type>pom</type>
				<scope>import</scope>
				<version>2.25.60</version>
			</dependency>
		</dependencies>
	</dependencyManagement>
	
			<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>s3</artifactId>
		</dependency>
		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>apache-client</artifactId>
		</dependency>
		<dependency>
			<groupId>software.amazon.awssdk.crt</groupId>
			<artifactId>aws-crt</artifactId>
			<version>0.29.20</version>
		</dependency>
		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>s3-transfer-manager</artifactId>
		</dependency>

AWS Java SDK version used

2.25.60

JDK version used

17

Operating System and version

windows11
@chetanyachopra22 chetanyachopra22 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Oct 9, 2024
@debora-ito
Copy link
Member

What is the actual endpoint used in the code below, is it an aws s3 endpoint or a third-party?

return S3AsyncClient
                .crtBuilder()
                .credentialsProvider(StaticCredentialsProvider.create(credentials))
                .region(Region.EU_WEST_1)
                .endpointOverride(new URI(endpoint))
                .forcePathStyle(true)
                .build();

@debora-ito debora-ito added response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. crt-client p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Oct 17, 2024
@debora-ito debora-ito self-assigned this Oct 17, 2024
@chetanyachopra22
Copy link
Author

Correct, I am using 3rd party Dell on prem object store ie S3 compatible storage.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. label Oct 21, 2024
@debora-ito
Copy link
Member

Do you see the same error when the s3 endpoint is used directly?

As a reminder, the AWS SDKs don't support third-party use cases. If the issue is on the third-party side you need to contact their support team.

@debora-ito debora-ito added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days. label Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@debora-ito debora-ito

Labels
bug This issue is a bug. crt-client p2 This is a standard priority issue response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 10 days.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@debora-ito @chetanyachopra22