[Investigation - MacOS] Seeing a ~3x (to 10x) performance difference between 2 versions of sample apps (Java EE vs Quarkus)

[jwmatthews]

Is there an existing issue for this?

• I have searched the existing issues

Konveyor version

:latest as of 2/21/2024

Priority

Undefined (Default)

Current Behavior

We are observing performance differences we do not understand as we build up a library of sample apps for Java EE to Quarkus migrations. All testing/comments in this issue relate to what we observe on various Mac OS apple silicon laptops at least M1 Max and M2.

We are seeing that analyzing a migrated version of the sample app from Java EE to Quarkus will take anywhere from 3x to 10x longer than the original Java EE version.

This work is ultimately being done to populate a library of sample apps and analysis reports for work under Kai at https://github.com/konveyor-ecosystem/kai/blob/main/samples/fetch_analyze_apps.py

As we build up this library of information we are focused on rerunning and storing analysis reports from an application at it's initial state (Java EE) and then also re-running after the application has been migrated to Quarkus. The discrepancy is that we see a huge performance difference and are unsure why the change is so dramatic.

Trying to understand why analysis on the Quarkus branch is sometimes 10X more expensive in the gist:
https://gist.github.com/savitharaghunathan/533fa71fb7dba3e76609f98647cd53bc

(Another data point from John running on M1 Max same script: https://gist.github.com/jwmatthews/c0b8d317b4a361c5e874dd24901d5ca7)

See this comment for a more extreme observation where we saw a 10x difference.

For example using this CMT application:

• CMT - 'main' branch: 2m9.854s : 94 files and ~84kb src content
• CMT - 'quarkus' branch: 25m3.627s: 103 files and ~120kb src content

'main' branch

Analyzing 'sample_repos/cmt', will write output to '/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial'
INFO[0000] running source code analysis                  args="--provider-settings=/opt/input/config/settings.json --output-file=/opt/output/output.yaml --context-lines=100 --rules=/opt/rulesets/ --dep-label-selector=(!konveyor.io/dep-source=open-source) --verbose=4 --label-selector=((konveyor.io/target=quarkus || konveyor.io/target=jakarta-ee || konveyor.io/target=jakarta-ee8+ || konveyor.io/target=jakarta-ee9+ || konveyor.io/target=cloud-readiness) && konveyor.io/source) || (discovery)" input=/Users/sraghuna/local_dev/kai/samples/sample_repos/cmt log=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial/analysis.log output=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial volumes="{\"/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial\":\"/opt/output\",\"/Users/sraghuna/local_dev/kai/samples/sample_repos/cmt\":\"/opt/input/source\",\"/var/folders/bj/s4n2xcqd0293c54fzjdtpcr00000gn/T/analyze-config-1208938299\":\"/opt/input/config\"}"
INFO[0000] generating analysis log in file               file=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial/analysis.log
INFO[0073] running dependency analysis                   args="--provider-settings=/opt/input/config/settings.json --output-file=/opt/output/output.yaml --context-lines=100 --rules=/opt/rulesets/ --dep-label-selector=(!konveyor.io/dep-source=open-source) --verbose=4 --label-selector=((konveyor.io/target=quarkus || konveyor.io/target=jakarta-ee || konveyor.io/target=jakarta-ee8+ || konveyor.io/target=jakarta-ee9+ || konveyor.io/target=cloud-readiness) && konveyor.io/source) || (discovery)" input=/Users/sraghuna/local_dev/kai/samples/sample_repos/cmt log=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial/dependency.log output=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial
INFO[0073] generating dependency log in file             file=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial/dependency.log
INFO[0127] generating static report                      args="/usr/local/bin/js-bundle-generator --analysis-output-list=/opt/output/output.yaml --deps-output-list=/opt/output/dependencies.yaml --output-path=/usr/local/static-report/output.js --application-name-list=cmt && cp -r /usr/local/static-report /opt/output" output=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial
INFO[0129] Static report created. Access it at this URL:  URL="file:/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/initial/static-report/index.html"

real    2m9.854s
user    0m0.237s
sys     0m0.258s

'quarkus' branch

Analyzing 'sample_repos/cmt', will write output to '/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved'
INFO[0000] running source code analysis                  args="--provider-settings=/opt/input/config/settings.json --output-file=/opt/output/output.yaml --context-lines=100 --rules=/opt/rulesets/ --dep-label-selector=(!konveyor.io/dep-source=open-source) --verbose=4 --label-selector=((konveyor.io/target=quarkus || konveyor.io/target=jakarta-ee || konveyor.io/target=jakarta-ee8+ || konveyor.io/target=jakarta-ee9+ || konveyor.io/target=cloud-readiness) && konveyor.io/source) || (discovery)" input=/Users/sraghuna/local_dev/kai/samples/sample_repos/cmt log=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved/analysis.log output=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved volumes="{\"/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved\":\"/opt/output\",\"/Users/sraghuna/local_dev/kai/samples/sample_repos/cmt\":\"/opt/input/source\",\"/var/folders/bj/s4n2xcqd0293c54fzjdtpcr00000gn/T/analyze-config-2551083917\":\"/opt/input/config\"}"
INFO[0000] generating analysis log in file               file=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved/analysis.log
INFO[0793] running dependency analysis                   args="--provider-settings=/opt/input/config/settings.json --output-file=/opt/output/output.yaml --context-lines=100 --rules=/opt/rulesets/ --dep-label-selector=(!konveyor.io/dep-source=open-source) --verbose=4 --label-selector=((konveyor.io/target=quarkus || konveyor.io/target=jakarta-ee || konveyor.io/target=jakarta-ee8+ || konveyor.io/target=jakarta-ee9+ || konveyor.io/target=cloud-readiness) && konveyor.io/source) || (discovery)" input=/Users/sraghuna/local_dev/kai/samples/sample_repos/cmt log=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved/dependency.log output=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved
INFO[0793] generating dependency log in file             file=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved/dependency.log
INFO[0889] generating static report                      args="/usr/local/bin/js-bundle-generator --analysis-output-list=/opt/output/output.yaml --deps-output-list=/opt/output/dependencies.yaml --output-path=/usr/local/static-report/output.js --application-name-list=cmt && cp -r /usr/local/static-report /opt/output" output=/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved
INFO[0892] Static report created. Access it at this URL:  URL="file:/Users/sraghuna/local_dev/kai/samples/analysis_reports/cmt/solved/static-report/index.html"

real    25m3.627s
user    0m0.263s
sys     0m0.281s

Expected Behavior

We expected to see performance numbers scale in proportion based on number of files/size of source content, so for CMT we observe:

• roughly ~10% more files and ~50% more bytes of data to scan

so we would assume the performance would be at a scale of maybe

• JavaEE is 2 mins
• Quarkus is 3 mins

How Reproducible

Always (Default)

Steps To Reproduce

We can break down the behavior to a single application such as CMT to aid as a reproducer.
https://github.com/konveyor-ecosystem/cmt.git

On a M1 Max we observed

• CMT 'main': 1m43.994s
• CMT 'quarkus: 6m50.406s

Below is how we are running this

1. Use this script to create podman machine VM on Mac OS (apple silicon): https://github.com/konveyor-ecosystem/kai/blob/main/samples/macos/restart_podman_machine.sh
2. Use this script to obtain latest Kantra binary: https://github.com/konveyor-ecosystem/kai/blob/main/samples/macos/get_latest_kantra_cli.sh
3. git clone https://github.com/konveyor-ecosystem/cmt.git
4. git checkout main
5. Run analyze_cmt.sh
6. git checkout quarkus
7. Run analyze_cmt.sh

Use the below analyze_cmt.sh to run the kantra analysis

$ cat analyze_cmt.sh 
SOURCE_DIR=cmt
OUTDIR=$PWD/analysis_reports/${SOURCE_DIR}
mkdir -p $OUTDIR
time ./bin/kantra analyze -i $PWD/sample_repos/$SOURCE_DIR -t "quarkus" -t "jakarta-ee" -t "jakarta-ee8+" -t "jakarta-ee9+" -t "cloud-readiness" -o $OUTDIR --overwrite

Environment

- OS: MacOS
- Arch: Apple Silicon, arm64

Anything else?

No response

[konveyor-ci-bot]

This issue is currently awaiting triage.
If contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members.

[jwmatthews]

Another data point from a run on M1Max, seeing this script (https://github.com/konveyor-ecosystem/kai/blob/main/samples/fetch_analyze_apps.py) take ~2 hours 45 minutes to complete to scan ~20 runs of 10 apps with an initial branch and then the solved branch.

https://gist.github.com/jwmatthews/4027954eef0ca53a218a05a184f06773

For most of the runs the branch which has been migrated to quarkus takes a lot longer to scan.

[pranavgaikwad]

@jwmatthews Let me know how things are improved for you when you get a chance. We made a handful performance fixes last few weeks, wondering whether things can be improved still.

[jwmatthews]

Below is a sample of what I see when I compare differences of Java EE vs Quarkus on a single application analysis.
This is using Kantra build as of March 19 2024. (MacOS M1Max arm64)

A difference of 22 seconds vs 455 seconds, so a ~20x difference in the sample application below.

Java EE

real	0m22.107s
user	0m0.114s
sys	0m0.149s
./example_analyze_single_app_with_custom_rules_tasks_qute.sh  0.12s user 0.16s system 1% cpu 22.152 total

Quarkus

real	7m35.338s
user	0m0.129s
sys	0m0.151s
./example_analyze_single_app_with_custom_rules_tasks_qute.sh  0.14s user 0.17s system 0% cpu 7:35.36 total

Example app

1. mkdir ./sample_apps
2. cd ./sample_apps
3. git clone https://github.com/konveyor-ecosystem/tasks-qute.git
4. cd ..

Example script

$ cat example_analyze_single_app_with_custom_rules_tasks_qute.sh 
#!/usr/bin/env bash

SOURCE_DIR="tasks-qute"

SOURCE_ONLY=""
# If you want to run with source only uncomment below
# SOURCE_ONLY="-m source-only"

# Choose to either analyze the initial or solved branch
# Then comment out/in the appropriate below
#BRANCH="main"
#OUTDIR=${PWD}/tmp/${SOURCE_DIR}/initial

BRANCH="quarkus"
OUTDIR=${PWD}/tmp/${SOURCE_DIR}/solved

# ####
# Ensure we are on the expected branch before analysis.
# We are typically working with 2 branches an initial/solved
# It's been a common problem to forget which and create invalid analysis runs
# ####
pushd .
cd "${PWD}"/sample_repos/"${SOURCE_DIR}" || exit
git checkout "${BRANCH}"
popd || exit

mkdir -p "${OUTDIR}"
time ./bin/kantra analyze -i "${PWD}"/sample_repos/"${SOURCE_DIR}" "${SOURCE_ONLY}" -t "quarkus" -t "jakarta-ee" -t "jakarta-ee8+" -t "jakarta-ee9+" -t "cloud-readiness" --rules ./custom_rules -o "${OUTDIR}" --overwrite

[pranavgaikwad]

@jwmatthews I think I can say with confidence that the 7m runtime you see will not grow linearly with size of the project. I think for a quarkus project, there's about 3m constant time to fetch dependencies that is not going away. Since you are running in full mode, it will always be there. I would be more interested in looking at windup's runtime on the same app. Note that even windup doesn't pull deps itself, so we cannot really compare 1:1.

[jwmatthews]

Thanks @pranavgaikwad, here is another data point that agrees with what you shared, i.e. ~7m is likely for the fetch of quarkus dependencies.

Ran on a larger application

Sample App: https://github.com/jmle/monolith.git

• Java EE branch (master): 23 minutes. (22m58.037s)
• Quarkus branch (quarkus): 30 minutes (29m48.767s)