Re-enable arm64 and x64 Mono fullAOT llvm and mini jobs #90427
Comments
|
Tagging subscribers to this area: @directhex Issue DetailsRefer to #86324 (comment), to re-enable the disabled lane.
|
ghost
added
in-pr
kotlarmilos
changed the title
|
Mono fullAOT linux runtime tests now utilize the new Mariner image. This image does not include binutils, and as a result, GNU One option is to utilize For linux-arm64, either option would require a cross-compiled toolchain for x64 targeting arm64. /cc: @directhex @lambdageek |
|
After several attempts to enable fullAOT llvm runtime tests on the extra-platforms, the job consistently hits OOM issue after 2h. To simplify the process, the job compiles only the The CI is using an azure instance with 16gb, docker on top of that is limited to less, probably 8gb. At peak, the AOT compilation takes about 16gb to AOT compile the @steveisok @SamMonoRT While we work on resolving the OOM issue, I would appreciate hearing your thoughts on temporarily disabling the affected jobs to ensure a consistently green CI. |
|
I'm fine with disabling for the time being. I wonder if requiring so much memory is a sign we can improve our aot compiler utilization? |
Yes, let's create a tracking issue so that we can brainstorm our next steps. Additionally, I'll temporarily disable the failing jobs. |
kotlarmilos
changed the title
kotlarmilos
added
tracking
The biarch image is currently unused in our ci builds. Use of the biarch image was added in #91019 but the job that used it was disabled for arm64 in #92057. #90427 tracks adding back the arm64 jobs. The arm64 fullaot job was added in #55362 which explains: > We don't ship any products using FullAOT on Linux x64/arm64, but this compilation mode is used for iOS, FullAOT-related issues that affect iOS are likely to be caught by FullAOT on Linux x64/arm64, and it is a lot easier to build, develop, and debug on desktop OSes. If the arm64 job is enabled again in the future, it should be made to use a new entry in `pipeline-with-resources.yml`, instead of modifying the image used for all other linux_arm64 jobs. Fixes #101707
The biarch image is currently unused in our ci builds. Use of the biarch image was added in dotnet#91019 but the job that used it was disabled for arm64 in dotnet#92057. dotnet#90427 tracks adding back the arm64 jobs. The arm64 fullaot job was added in dotnet#55362 which explains: > We don't ship any products using FullAOT on Linux x64/arm64, but this compilation mode is used for iOS, FullAOT-related issues that affect iOS are likely to be caught by FullAOT on Linux x64/arm64, and it is a lot easier to build, develop, and debug on desktop OSes. If the arm64 job is enabled again in the future, it should be made to use a new entry in `pipeline-with-resources.yml`, instead of modifying the image used for all other linux_arm64 jobs. Fixes dotnet#101707
The biarch image is currently unused in our ci builds. Use of the biarch image was added in dotnet#91019 but the job that used it was disabled for arm64 in dotnet#92057. dotnet#90427 tracks adding back the arm64 jobs. The arm64 fullaot job was added in dotnet#55362 which explains: > We don't ship any products using FullAOT on Linux x64/arm64, but this compilation mode is used for iOS, FullAOT-related issues that affect iOS are likely to be caught by FullAOT on Linux x64/arm64, and it is a lot easier to build, develop, and debug on desktop OSes. If the arm64 job is enabled again in the future, it should be made to use a new entry in `pipeline-with-resources.yml`, instead of modifying the image used for all other linux_arm64 jobs. Fixes dotnet#101707
Description
Currently, the Mono fullAOT LLVM jobs are encountering failures and don't provide any test coverage. Specifically, the linux-arm64 job must build the cross-compiler using an image which incorporates both x64 and arm64 rootfs. Additionally, the aot-compiler.c utilizes GNU
aswhich is not available since we don't have GNU binutils on the Mariner.To resolve this issue, we have to fix the mono-aot-cross build for linux-arm64 to use target rootfs when building the runtime and host rootfs when building the compiler. After that, the aot-compiler.c has to be updated to use
llvm-asorclanginstead of GNUas.Additionally, the AOT compiler with llvm 16 toolchain hits OOM issue with CBL_Mariner images.
Tasks
Initial discussion can be found at #86324 (comment).
Disabled tests
Upon the initial setup of the jobs, the following tests require triage.
FullAOT mini
AOT compilation failure:
Execution failure:
FullAOT LLVM
AOT compilation failure:
Execution failure:
The text was updated successfully, but these errors were encountered: