I am getting multiple exceptions while fetching data from dynamo db suddenly.

[danyal-careem]

Describe the bug

The issue is fetching the data from dynamo db based on user Id. which throws two exceptions. It was working fine till 20th of August suddenly getting those exceptions which getting higher day by day. One more exception we had "Error thrown from TransformingAsyncResponseHandler#onError, ignoring" which was resolved after updating SDK version.

1. java.lang.InterruptedException: null at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.loadPartition(DDBExpressionUserInfoRepository.java:225)

2. java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.loadPartition(DDBExpressionUserInfoRepository.java:225)

JDK image is: eclipse-temurin:22-jdk
SDK version: 2.18.21 (which resolved error, thrown from TransformingAsyncResponseHandler issue)

Code snippet:

private Map<String, Object> loadPartition(Integer userId, Collection<String> subjects) {
    Map<String, Object> info = new HashMap<>();
    // building the primary key for each item to be requested
    List<Map<String, AttributeValue>> keys = subjects.stream()
        // we add the subject as an empty map entry to signify a lookup has been done
        .peek(subject -> info.put(subject, null))
        .map(subject -> Map.of(
            "user_id", AttributeValue.builder().n(Integer.toString(userId)).build(),
                SUBJECT_KEY, AttributeValue.builder().s(subject).build())
        )
        .collect(Collectors.toList());

    try {

      BatchGetItemResponse response = this.ddbClient.batchGetItem(
              BatchGetItemRequest.builder()
                      .requestItems(
                              Map.of(this.tableName, KeysAndAttributes.builder().keys(keys).build())
                      )
                      .build()
      ).get();

      Map<String, KeysAndAttributes> unprocessed;
      do {
        response.responses().get(this.tableName)
            .forEach(item -> {
              var subject = item.get(SUBJECT_KEY).s();
              item.forEach(
                  (key, attributeValue) -> {
                    if (isDynamoPayload(key)) {
                      loadDynamoItem(info, subject, key, attributeValue);
                    }
                  }
              );
            });

        // if 16MB or 100 items reached, the response will be truncated
        // the rest can be fetched by loading for unprocessed keys
        unprocessed = response.unprocessedKeys();
        if (!unprocessed.isEmpty()) {
          response = this.ddbClient.batchGetItem(
              BatchGetItemRequest.builder().requestItems(unprocessed).build()
          ).get();
        }
      }
      while (!unprocessed.isEmpty());

      return info;
    } catch (Exception e) {
      log.warn(USER_ACCESS_FAILURE, userId, e);
      circuitBreakerByUserID.put(userId, true);
      Thread.currentThread().interrupt();
      return null;
    }

Expected Behavior

it has to return data based on userId as it was working before.

No Throttle in db.

Current Behavior

Getting multiple exceptions.

Reproduction Steps

1. Request any userId which present in dynamo db with data.
2. Send data userId and subjects using SDK batchGetItem.
3. It throws above exception.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

eclipse-temurin:22-jdk

JDK version used

22

Operating System and version

eclipse-temurin

[debora-ito]

@danyal-careem can you share the full stacktrace with the errors?

You mentioned the errors started to show up suddenly in August, was there any change in code, SDK version or environment configuration that could have trigger the errors?

[danyal-careem]

No change in code related to that let me share a full trace with you.
java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException: HTTP request execution did not complete before the specified timeout configuration: 200 millis at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.getUserInfo(DDBExpressionUserInfoRepository.java:115) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.getUserInfo(DDBExpressionUserInfoRepository.java:75) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.getUserInfo(DDBExpressionUserInfoRepository.java:68) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository$$FastClassBySpringCGLIB$$8ada38b3.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.cloud.sleuth.instrument.annotation.SleuthInterceptor.invoke(SleuthAdvisorConfig.java:188) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository$$EnhancerBySpringCGLIB$$ce392517.getUserInfo(<generated>) at com.careem.launcher.service.expression.UserDataProvider.load(UserDataProvider.java:63) at com.careem.launcher.service.expression.solver.UserExpressionObject.isInBrazeCampaign(UserExpressionObject.java:109) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:139) at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:139) at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:55) at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:386) at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:92) at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:112) at org.springframework.expression.spel.ast.InlineMap.getValueInternal(InlineMap.java:128) at org.springframework.expression.spel.ast.MethodReference.getArguments(MethodReference.java:164) at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:94) at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:117) at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:376) at com.careem.launcher.service.ExpressionEvaluationService.evaluateTemplate(ExpressionEvaluationService.java:151) at com.careem.launcher.service.ExpressionEvaluationService.lambda$evaluateTemplates$5(ExpressionEvaluationService.java:137) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194) at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1858) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611) at com.careem.launcher.service.ExpressionEvaluationService.evaluateTemplates(ExpressionEvaluationService.java:137) at com.careem.launcher.service.ExpressionEvaluationService.lambda$evaluateTileTemplates$1(ExpressionEvaluationService.java:84) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:265) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:702) at com.careem.launcher.service.TileService.renderTiles(TileService.java:630) at com.careem.launcher.service.TileService.lambda$getResolvedServiceTiles$20(TileService.java:398) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106) at com.dynatrace.agent.introspection.reactor3.TracingReactorSubscriber.lambda$onNext$0(TracingReactorSubscriber.java:36) at com.dynatrace.agent.introspection.reactor3.TracingReactorSubscriber.withActivePath(TracingReactorSubscriber.java:62) at com.dynatrace.agent.introspection.reactor3.TracingReactorSubscriber.onNext(TracingReactorSubscriber.java:36) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:89) at com.dynatrace.agent.introspection.reactor3.TracingReactorSubscriber.lambda$onNext$0(TracingReactorSubscriber.java:36) at com.dynatrace.agent.introspection.reactor3.TracingReactorSubscriber.withActivePath(TracingReactorSubscriber.java:62) at com.dynatrace.agent.introspection.reactor3.TracingReactorSubscriber.onNext(TracingReactorSubscriber.java:36) at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181) at org.springframework.cloud.sleuth.instrument.reactor.ReactorSleuth.lambda$null$6(ReactorSleuth.java:324) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1570) Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException: HTTP request execution did not complete before the specified timeout configuration: 200 millis at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223) at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182) at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$null$0(MakeAsyncHttpRequestStage.java:103) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:240) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:163) at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ... 3 common frames omitted Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: HTTP request execution did not complete before the specified timeout configuration: 200 millis Caused by: software.amazon.awssdk.http.nio.netty.internal.FutureCancelledException: software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException: HTTP request execution did not complete before the specified timeout configuration: 200 millis at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.lambda$null$3(NettyRequestExecutor.java:136) at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at io.netty.util.concurrent.PromiseTask.run(PromiseTask.java:106) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ... 1 common frames omitted Caused by: software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException: HTTP request execution did not complete before the specified timeout configuration: 200 millis at software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException$BuilderImpl.build(ApiCallAttemptTimeoutException.java:97) at software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException.create(ApiCallAttemptTimeoutException.java:38) at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$setupAttemptTimer$8(MakeAsyncHttpRequestStage.java:228) at software.amazon.awssdk.core.internal.http.timers.AsyncTimeoutTask.run(AsyncTimeoutTask.java:52) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ... 1 common frames omitted

[debora-ito]

software.amazon.awssdk.core.exception.ApiCallAttemptTimeoutException: HTTP request execution did not complete before the specified timeout configuration: 200 millis

The message is saying that the request execution didn't complete before the configured 200ms timeout. You can try to increase the timeout in ClientOverrideConfiguration or reduce the number of items in the batch request.

Also, note that if there was no code change or SDK version change, the rootcause probably isn't in the SDK, the SDK doesn't change behavior if the code is the same. In these cases you should look into any network, transaction spike or environment configuration changes that could have affected the application run.

[danyal-careem]

Sorry, the Stacktrace I shared from dev which is clearly mentioned the timeout but that not the case in Prod.
For Prod I am getting two different exceptions and I requested same data directly to dynamo db as well which is fetching fine.

java.lang.InterruptedException: null at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.loadPartition(DDBExpressionUserInfoRepository.java:225) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.lambda$getAllUserInfo$4(DDBExpressionUserInfoRepository.java:166) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.getAllUserInfo(DDBExpressionUserInfoRepository.java:165) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository$$FastClassBySpringCGLIB$$8ada38b3.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at io.micrometer.core.aop.CountedAspect.interceptAndRecord(CountedAspect.java:206) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at io.micrometer.core.aop.TimedAspect.processWithTimer(TimedAspect.java:221) at io.micrometer.core.aop.TimedAspect.perform(TimedAspect.java:196) at io.micrometer.core.aop.TimedAspect.timedMethod(TimedAspect.java:188) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.cloud.sleuth.instrument.annotation.SleuthInterceptor.invoke(SleuthAdvisorConfig.java:188) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository$$EnhancerBySpringCGLIB$$8460f40a.getAllUserInfo(<generated>) at com.careem.launcher.service.expression.UserDataProvider.prepopulateCache(UserDataProvider.java:114) at com.careem.launcher.service.ExpressionEvaluationService.newExpressionContext(ExpressionEvaluationService.java:204) at com.careem.launcher.service.ExpressionContextProvider.enrichRequestGetContext(ExpressionContextProvider.java:40) at com.careem.launcher.service.TileService.getHomeScreen(TileService.java:123) at com.careem.launcher.service.TileService$$FastClassBySpringCGLIB$$aa15cb68.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.cloud.sleuth.instrument.annotation.ReactorSleuthMethodInvocationProcessor.proceedUnderReactorSpan(ReactorSleuthMethodInvocationProcessor.java:79) at org.springframework.cloud.sleuth.instrument.annotation.ReactorSleuthMethodInvocationProcessor.process(ReactorSleuthMethodInvocationProcessor.java:57) at org.springframework.cloud.sleuth.instrument.annotation.SleuthInterceptor.invoke(SleuthAdvisorConfig.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.careem.launcher.service.TileService$$EnhancerBySpringCGLIB$$e47fb65d.getHomeScreen(<generated>) at com.careem.launcher.api.TilesController.lambda$getHomeScreen$11(TilesController.java:160) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:89) at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181) at org.springframework.cloud.sleuth.instrument.reactor.ReactorSleuth.lambda$null$6(ReactorSleuth.java:324) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1570)

Second one is:

java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.loadPartition(DDBExpressionUserInfoRepository.java:225) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.lambda$getAllUserInfo$4(DDBExpressionUserInfoRepository.java:166) at java.base/java.lang.Iterable.forEach(Iterable.java:75) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.getAllUserInfo(DDBExpressionUserInfoRepository.java:165) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository$$FastClassBySpringCGLIB$$8ada38b3.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at io.micrometer.core.aop.CountedAspect.interceptAndRecord(CountedAspect.java:206) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at io.micrometer.core.aop.TimedAspect.processWithTimer(TimedAspect.java:221) at io.micrometer.core.aop.TimedAspect.perform(TimedAspect.java:196) at io.micrometer.core.aop.TimedAspect.timedMethod(TimedAspect.java:188) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.cloud.sleuth.instrument.annotation.SleuthInterceptor.invoke(SleuthAdvisorConfig.java:188) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository$$EnhancerBySpringCGLIB$$b43d508f.getAllUserInfo(<generated>) at com.careem.launcher.service.expression.UserDataProvider.prepopulateCache(UserDataProvider.java:114) at com.careem.launcher.service.ExpressionEvaluationService.newExpressionContext(ExpressionEvaluationService.java:204) at com.careem.launcher.service.ExpressionContextProvider.enrichRequestGetContext(ExpressionContextProvider.java:40) at com.careem.launcher.service.TileService.getHomeScreen(TileService.java:123) at com.careem.launcher.service.TileService$$FastClassBySpringCGLIB$$aa15cb68.invoke(<generated>) at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.cloud.sleuth.instrument.annotation.ReactorSleuthMethodInvocationProcessor.proceedUnderReactorSpan(ReactorSleuthMethodInvocationProcessor.java:79) at org.springframework.cloud.sleuth.instrument.annotation.ReactorSleuthMethodInvocationProcessor.process(ReactorSleuthMethodInvocationProcessor.java:57) at org.springframework.cloud.sleuth.instrument.annotation.SleuthInterceptor.invoke(SleuthAdvisorConfig.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) at com.careem.launcher.service.TileService$$EnhancerBySpringCGLIB$$e47fb65d.getHomeScreen(<generated>) at com.careem.launcher.api.TilesController.lambda$getHomeScreen$11(TilesController.java:160) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) at org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onNext(ScopePassingSpanSubscriber.java:89) at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:181) at org.springframework.cloud.sleuth.instrument.reactor.ReactorSleuth.lambda$null$6(ReactorSleuth.java:324) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1570) Caused by: software.amazon.awssdk.core.exception.SdkClientException: null at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:217) at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:288) at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:227) at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80) at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294) at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73) at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49) at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbAsyncClient.lambda$batchGetItem$5(DefaultDynamoDbAsyncClient.java:485) at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341) at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbAsyncClient.batchGetItem(DefaultDynamoDbAsyncClient.java:484) at com.careem.launcher.data.repo.ddb.DDBExpressionUserInfoRepository.loadPartition(DDBExpressionUserInfoRepository.java:219) ... 69 common frames omitted Caused by: software.amazon.awssdk.core.exception.SdkInterruptedException: null at software.amazon.awssdk.core.internal.http.InterruptMonitor.checkInterrupted(InterruptMonitor.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApplyTransactionIdStage.execute(ApplyTransactionIdStage.java:43) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApplyTransactionIdStage.execute(ApplyTransactionIdStage.java:29) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:213) ... 80 common frames omitted