[BUG] Error "Failed to get metadata for table" with connector Confluent Cloud

[Carlos4ndresh]

Describe the bug
Following: https://aws.amazon.com/blogs/business-intelligence/visualize-confluent-data-in-amazon-quicksight-using-amazon-athena/, when reaching the stage of configuring the Athena Connector, and discovering the tables, this appears on the UI:

To Reproduce
Steps to reproduce the behavior:

1. Follow https://aws.amazon.com/blogs/business-intelligence/visualize-confluent-data-in-amazon-quicksight-using-amazon-athena/, until step 18.
2. Click on 'Amazon Athena -> Data sources -> ' and then on the "Associated Database" that appears.
3. You'll see the error in the image
4. If you go to the lambda function CW logs, you'll see this stack trace or similar:

2024-05-29 15:07:51 4779d5fb-806f-4174-9405-a29663f26481 WARN  CompositeHandler:109 - Client's SerDe mis-matched with connector version:, attempt with lower version: '4'
java.lang.IllegalStateException: Expected field name token but got END_OBJECT
	at com.amazonaws.athena.connector.lambda.serde.BaseDeserializer.assertFieldName(BaseDeserializer.java:239) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.serde.v5.GetTableRequestSerDeV5$Deserializer.doRequestDeserialize(GetTableRequestSerDeV5.java:94) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.serde.v5.GetTableRequestSerDeV5$Deserializer.doRequestDeserialize(GetTableRequestSerDeV5.java:74) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.serde.v2.MetadataRequestDeserializer.doTypedDeserialize(MetadataRequestDeserializer.java:56) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.serde.v2.MetadataRequestDeserializer.doTypedDeserialize(MetadataRequestDeserializer.java:36) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.serde.DelegatingDeserializer.doDeserialize(DelegatingDeserializer.java:59) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.serde.DelegatingDeserializer.deserializeWithType(DelegatingDeserializer.java:49) ~[task/:?]
	at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:74) ~[task/:?]
	at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342) ~[task/:?]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4905) ~[task/:?]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3909) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:105) [task/:?]
	at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:925) [aws-lambda-java-runtime-0.2.0.jar:?]
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:268) [aws-lambda-java-runtime-0.2.0.jar:?]
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:207) [aws-lambda-java-runtime-0.2.0.jar:?]
	at lambdainternal.AWSLambda.main(AWSLambda.java:196) [aws-lambda-java-runtime-0.2.0.jar:?]
2024-05-29 15:07:52 4779d5fb-806f-4174-9405-a29663f26481 WARN  CompositeHandler:126 - handleRequest: Completed with an exception.
java.lang.NullPointerException: null value in entry: dataFormat=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33) ~[task/:?]
	at com.google.common.collect.ImmutableMapEntry.<init>(ImmutableMapEntry.java:54) ~[task/:?]
	at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:345) ~[task/:?]
	at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:454) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.data.SchemaBuilder.addMetadata(SchemaBuilder.java:278) ~[task/:?]
	at com.amazonaws.athena.connectors.kafka.KafkaMetadataHandler.getSchema(KafkaMetadataHandler.java:443) ~[task/:?]
	at com.amazonaws.athena.connectors.kafka.KafkaMetadataHandler.doGetTable(KafkaMetadataHandler.java:267) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.handlers.MetadataHandler.resolveDoGetTableImplementation(MetadataHandler.java:336) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.handlers.MetadataHandler.doHandleRequest(MetadataHandler.java:273) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:154) ~[task/:?]
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:122) [task/:?]
	at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:925) [aws-lambda-java-runtime-0.2.0.jar:?]
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:268) [aws-lambda-java-runtime-0.2.0.jar:?]
	at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:207) [aws-lambda-java-runtime-0.2.0.jar:?]
	at lambdainternal.AWSLambda.main(AWSLambda.java:196) [aws-lambda-java-runtime-0.2.0.jar:?]
null value in entry: dataFormat=null: java.lang.NullPointerException
java.lang.NullPointerException: null value in entry: dataFormat=null
	at com.google.common.collect.CollectPreconditions.checkEntryNotNull(CollectPreconditions.java:33)
	at com.google.common.collect.ImmutableMapEntry.<init>(ImmutableMapEntry.java:54)
	at com.google.common.collect.ImmutableMap.entryOf(ImmutableMap.java:345)
	at com.google.common.collect.ImmutableMap$Builder.put(ImmutableMap.java:454)
	at com.amazonaws.athena.connector.lambda.data.SchemaBuilder.addMetadata(SchemaBuilder.java:278)
	at com.amazonaws.athena.connectors.kafka.KafkaMetadataHandler.getSchema(KafkaMetadataHandler.java:443)
	at com.amazonaws.athena.connectors.kafka.KafkaMetadataHandler.doGetTable(KafkaMetadataHandler.java:267)
	at com.amazonaws.athena.connector.lambda.handlers.MetadataHandler.resolveDoGetTableImplementation(MetadataHandler.java:336)
	at com.amazonaws.athena.connector.lambda.handlers.MetadataHandler.doHandleRequest(MetadataHandler.java:273)
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:154)
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:122)

Expected behavior
DS configuration should be successful and data from the Kafka Topics should be queriable and discovered by the connector/lambda function

Connector Details (please complete the following information):

• Version: serverlessrepo:semanticVersion 2024.20.2
• Name: AthenaKafkaConnector