aws-athena-federation-sdk build failed

[may215]

Hi,
I tried to build on mac(M1) the aws-athena-federation-sdk and build failed:

Tests run: 188, Failures: 0, Errors: 93, Skipped: 0

Java Version:
java 17.0.2 2022-01-18 LTS
Java(TM) SE Runtime Environment (build 17.0.2+8-LTS-86)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.2+8-LTS-86, mixed mode, sharing)

Maven version:
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /opt/homebrew/Cellar/maven/3.8.6/libexec
Java version: 18.0.1.1, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/18.0.1.1/libexec/openjdk.jdk/Contents/Home
Default locale: en_IL, platform encoding: UTF-8
OS name: "mac os x", version: "12.0.1", arch: "aarch64", family: "mac"

Action:
mvn clean install

After run mvn clean install inside athena-federation-integ-test folder.
I am getting those errors:

[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] com.amazonaws.athena.connectors.cloudwatch.CloudwatchMetadataHandlerTest.doGetSplits
[ERROR]   Run 1: CloudwatchMetadataHandlerTest.doGetSplits:367 » NoClassDefFound Could not init...
[ERROR]   Run 2: CloudwatchMetadataHandlerTest.tearDown:122 » IllegalState Allocator[ROOT] clos...
[INFO]
[ERROR] com.amazonaws.athena.connectors.cloudwatch.CloudwatchMetadataHandlerTest.doGetTableLayout
[ERROR]   Run 1: CloudwatchMetadataHandlerTest.doGetTableLayout:327 » NoClassDefFound Could not...
[ERROR]   Run 2: CloudwatchMetadataHandlerTest.tearDown:122 » IllegalState Allocator[ROOT] clos...
[INFO]
[ERROR] com.amazonaws.athena.connectors.cloudwatch.CloudwatchRecordHandlerTest.doReadRecordsNoSpill
[ERROR]   Run 1: CloudwatchRecordHandlerTest.doReadRecordsNoSpill:200 » NoClassDefFound Could n...
[ERROR]   Run 2: CloudwatchRecordHandlerTest.tearDown:189 » IllegalState Allocator[ROOT] closed...
[INFO]
[ERROR] com.amazonaws.athena.connectors.cloudwatch.CloudwatchRecordHandlerTest.doReadRecordsSpill
[ERROR]   Run 1: CloudwatchRecordHandlerTest.doReadRecordsSpill:240 » ExceptionInInitializer
[ERROR]   Run 2: CloudwatchRecordHandlerTest.tearDown:189 » IllegalState Allocator[ROOT] closed...
[INFO]
[INFO]
[ERROR] Tests run: 7, Failures: 0, Errors: 4, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.742 s
[INFO] Finished at: 2022-06-12T16:42:09+03:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project athena-cloudwatch: There are test failures.
[ERROR]

Do I need to do something for this to pass?

[henrymai]

If those errors are coming from you running mvn clean install under the athena-federation-integ-test folder, something is very wrong, because those tests are not under that project directory.

com.amazonaws.athena.connectors.cloudwatch.CloudwatchMetadataHandlerTest and com.amazonaws.athena.connectors.cloudwatch.CloudwatchRecordHandlerTest are under the athena-cloudwatch directory.

Can you run this command: pwd after you get those failures and update this issue?

Also aside from the directory inconsistency, when you rerun, can you use the command as follows so that we can get the full stack traces?

mvn -X -DtrimStackTrace=false clean install

Additionally when you get those errors, if you scroll up, there should be a stack trace from during the run of the test. Can you make sure to copy paste or attach the full output of your terminal to include those stack traces?

[may215]

hi,
When I am buikding the athena-federation-integ-test folder it's successfully build. but, when I am buidling the aws-athena-federation-sdk, using mvn -X -DtrimStackTrace=false clean install I am getting the above tests errors

[henrymai]

when I am buidling the aws-athena-federation-sdk, using mvn -X -DtrimStackTrace=false clean install I am getting the above tests errors

Yes, I am asking you to provide the output of that command.

The output that you are showing right now is truncated.

For example:
Run 1: CloudwatchMetadataHandlerTest.doGetSplits:367 » NoClassDefFound Could not init...

Is an example of a truncated error message and there should be a more complete output further up in the console, otherwise the truncated messages don't really tell us anything that can help you further.

[may215]

This is the output of the logs:
https://docs.google.com/document/d/1LR8q32R-JnwumBACu0bT-KmReMUvPqK7/edit?usp=sharing&ouid=101806259760079365555&rtpof=true&sd=true

[henrymai]

From your logs, it looks like its an issue with Apache Arrow on your setup:

50185 java.lang.NoClassDefFoundError: Could not initialize class org.apache.arrow.memory.util.MemoryUtil
50186   at org.apache.arrow.memory.ArrowBuf.setZero(ArrowBuf.java:1161)
50187   at org.apache.arrow.vector.complex.StructVector.reallocValidityBuffer(StructVector.java:473)
50188   at org.apache.arrow.vector.complex.StructVector.setValueCount(StructVector.java:594)
50189   at org.apache.arrow.vector.VectorSchemaRoot.setRowCount(VectorSchemaRoot.java:240)
50190   at com.amazonaws.athena.connector.lambda.data.Block.setRowCount(Block.java:242)
50191   at com.amazonaws.athena.connector.lambda.handlers.UserDefinedFunctionHandlerTest.createUDFRequest(UserDefinedFunctionHandlerTest.java:232)
50192   at com.amazonaws.athena.connector.lambda.handlers.UserDefinedFunctionHandlerTest.testInvocationWithStructType(UserDefinedFunctionHandlerTest.java:139)
50193   at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
50194   at java.base/java.lang.reflect.Method.invoke(Method.java:577)
50195   at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
[...]
50221 Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.RuntimeException: Failed to initialize MemoryUtil. [in thread "main"]
50222   at org.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:130)
50223   at org.apache.arrow.memory.ArrowBuf.setZero(ArrowBuf.java:1161)
50224   at org.apache.arrow.vector.BaseFixedWidthVector.reAlloc(BaseFixedWidthVector.java:446)
50225   at org.apache.arrow.vector.BaseFixedWidthVector.handleSafe(BaseFixedWidthVector.java:836)
50226   at org.apache.arrow.vector.IntVector.setSafe(IntVector.java:210)
50227   at com.amazonaws.athena.connector.lambda.data.BlockUtils.setValue(BlockUtils.java:306)
50228   at com.amazonaws.athena.connector.lambda.security.BlockCryptoTest.test(BlockCryptoTest.java:61)

There might be a problem with the Apache Arrow version we are using and M1 Macs or it might be a problem with your JDK version.

I noticed that you are using JDK17 on your machine and a maven built using JDK18.

This project currently expects to be built and run using JDK11:

        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>

So you can try installing JDK11 on your machine and also installing a maven compatible with it.

If that doesn't help, then its an issue with the version of Arrow and your M1 Mac.

Not the same issue, but might be related: snowflakedb/snowflake-jdbc#533 (comment)