Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.IO.Compression.Brotli #86335

Closed
performanceautofiler bot opened this issue May 16, 2023 · 6 comments
Closed

Regressions in System.IO.Compression.Brotli #86335

performanceautofiler bot opened this issue May 16, 2023 · 6 comments
Assignees
@DrewScoggins
Labels
arch-x64 area-Infrastructure-coreclr os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline eead6a328cf3a7426e62d1441343b4b0e1f57d19
Compare 0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Compression.Brotli

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Decompress_WithState - Duration of single invocation 275.33 μs 290.84 μs 1.06 0.01 False
Decompress_WithoutState - Duration of single invocation 246.86 μs 260.12 μs 1.05 0.01 False
Decompress_WithoutState - Duration of single invocation 275.01 μs 290.06 μs 1.05 0.01 False
Decompress - Duration of single invocation 258.04 μs 271.42 μs 1.05 0.01 False
Decompress - Duration of single invocation 282.37 μs 297.75 μs 1.05 0.01 False
Decompress_WithState - Duration of single invocation 247.79 μs 262.28 μs 1.06 0.01 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Compression.Brotli*'

Payloads

Baseline
Compare

Histogram

System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.84272255291006 > 289.21183704769743.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.94069941552576 (T) = (0 -290639.580362342) / Math.Sqrt((107646.4212471317 / (20)) + (506031.5362730523 / (19))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (19) - 2, .025) and -0.05517818741065231 = (275441.2324192894 - 290639.580362342) / 275441.2324192894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 260.1242030555556 > 259.227968737793.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -62.00641912288732 (T) = (0 -260669.61401916188) / Math.Sqrt((37447.13882312735 / (20)) + (965191.1463435948 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.0562565098872357 = (246786.2792599409 - 260669.61401916188) / 246786.2792599409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.0647317129629 > 288.7385472341009.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -109.10396387586594 (T) = (0 -289757.7017100669) / Math.Sqrt((64328.792429948866 / (20)) + (301071.5115852465 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05362418781284854 = (275010.48766881146 - 289757.7017100669) / 275010.48766881146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.4164824270557 > 270.9308018344105.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.4011638108043 (T) = (0 -272065.7148001677) / Math.Sqrt((101261.5224972446 / (20)) + (441966.70041789534 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05388187749429565 = (258155.79583456717 - 272065.7148001677) / 258155.79583456717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 297.7458011287062 > 296.70168234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -148.62331465251418 (T) = (0 -297489.1681221059) / Math.Sqrt((103374.3582741498 / (20)) + (96150.73892757313 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05252066097941168 = (282644.4925511301 - 297489.1681221059) / 282644.4925511301 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 262.27939534722225 > 259.47930234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -76.42147528289804 (T) = (0 -260977.19148595305) / Math.Sqrt((73102.83454388827 / (20)) + (570478.22836798 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05544132342591581 = (247268.30918352964 - 260977.19148595305) / 247268.30918352964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels May 16, 2023
@EgorBo EgorBo changed the title [Perf] Linux/x64: 6 Regressions on 5/10/2023 6:18:38 PM Regressions in System.IO.Compression.Brotli May 16, 2023
@EgorBo EgorBo transferred this issue from dotnet/perf-autofiling-issues May 16, 2023
@ghost
Copy link

ghost commented May 16, 2023

Tagging subscribers to this area: @dotnet/area-system-io-compression
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline eead6a328cf3a7426e62d1441343b4b0e1f57d19
Compare 0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Compression.Brotli

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Decompress_WithState - Duration of single invocation 275.33 μs 290.84 μs 1.06 0.01 False
Decompress_WithoutState - Duration of single invocation 246.86 μs 260.12 μs 1.05 0.01 False
Decompress_WithoutState - Duration of single invocation 275.01 μs 290.06 μs 1.05 0.01 False
Decompress - Duration of single invocation 258.04 μs 271.42 μs 1.05 0.01 False
Decompress - Duration of single invocation 282.37 μs 297.75 μs 1.05 0.01 False
Decompress_WithState - Duration of single invocation 247.79 μs 262.28 μs 1.06 0.01 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Compression.Brotli*'

Payloads

Baseline
Compare

Histogram

System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.84272255291006 > 289.21183704769743.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.94069941552576 (T) = (0 -290639.580362342) / Math.Sqrt((107646.4212471317 / (20)) + (506031.5362730523 / (19))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (19) - 2, .025) and -0.05517818741065231 = (275441.2324192894 - 290639.580362342) / 275441.2324192894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 260.1242030555556 > 259.227968737793.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -62.00641912288732 (T) = (0 -260669.61401916188) / Math.Sqrt((37447.13882312735 / (20)) + (965191.1463435948 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.0562565098872357 = (246786.2792599409 - 260669.61401916188) / 246786.2792599409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.0647317129629 > 288.7385472341009.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -109.10396387586594 (T) = (0 -289757.7017100669) / Math.Sqrt((64328.792429948866 / (20)) + (301071.5115852465 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05362418781284854 = (275010.48766881146 - 289757.7017100669) / 275010.48766881146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.4164824270557 > 270.9308018344105.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.4011638108043 (T) = (0 -272065.7148001677) / Math.Sqrt((101261.5224972446 / (20)) + (441966.70041789534 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05388187749429565 = (258155.79583456717 - 272065.7148001677) / 258155.79583456717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 297.7458011287062 > 296.70168234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -148.62331465251418 (T) = (0 -297489.1681221059) / Math.Sqrt((103374.3582741498 / (20)) + (96150.73892757313 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05252066097941168 = (282644.4925511301 - 297489.1681221059) / 282644.4925511301 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 262.27939534722225 > 259.47930234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -76.42147528289804 (T) = (0 -260977.19148595305) / Math.Sqrt((73102.83454388827 / (20)) + (570478.22836798 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05544132342591581 = (247268.30918352964 - 260977.19148595305) / 247268.30918352964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: DrewScoggins
Labels:

area-System.IO.Compression, os-linux, arch-x64, untriaged, runtime-coreclr, PGO
Milestone: -

@EgorBo
Copy link
Member

EgorBo commented May 16, 2023

Seems to be regressed by #84676 @sbomer

@ViktorHofer
Copy link
Member

@DrewScoggins should this issue be moved to another area? Based on @EgorBo's comment above, the cause is likely unrelated to System.IO.Compression.Brotli?

@ViktorHofer ViktorHofer removed the untriaged New issue has not been triaged by the area owner label Jun 7, 2023
@ViktorHofer ViktorHofer added this to the 8.0.0 milestone Jun 7, 2023
@DrewScoggins
Copy link
Member

Yeah, we can just relabel this with the same label as the issue that @EgorBo linked.

@ghost
Copy link

ghost commented Jun 7, 2023

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline eead6a328cf3a7426e62d1441343b4b0e1f57d19
Compare 0b5f137ab0eea82f59d22da94a0b21f0bc598f16
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Compression.Brotli

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Decompress_WithState - Duration of single invocation 275.33 μs 290.84 μs 1.06 0.01 False
Decompress_WithoutState - Duration of single invocation 246.86 μs 260.12 μs 1.05 0.01 False
Decompress_WithoutState - Duration of single invocation 275.01 μs 290.06 μs 1.05 0.01 False
Decompress - Duration of single invocation 258.04 μs 271.42 μs 1.05 0.01 False
Decompress - Duration of single invocation 282.37 μs 297.75 μs 1.05 0.01 False
Decompress_WithState - Duration of single invocation 247.79 μs 262.28 μs 1.06 0.01 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Compression.Brotli*'

Payloads

Baseline
Compare

Histogram

System.IO.Compression.Brotli.Decompress_WithState(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.84272255291006 > 289.21183704769743.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.94069941552576 (T) = (0 -290639.580362342) / Math.Sqrt((107646.4212471317 / (20)) + (506031.5362730523 / (19))) is less than -2.026192463026769 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (19) - 2, .025) and -0.05517818741065231 = (275441.2324192894 - 290639.580362342) / 275441.2324192894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithoutState(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 260.1242030555556 > 259.227968737793.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -62.00641912288732 (T) = (0 -260669.61401916188) / Math.Sqrt((37447.13882312735 / (20)) + (965191.1463435948 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.0562565098872357 = (246786.2792599409 - 260669.61401916188) / 246786.2792599409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithoutState(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.0647317129629 > 288.7385472341009.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -109.10396387586594 (T) = (0 -289757.7017100669) / Math.Sqrt((64328.792429948866 / (20)) + (301071.5115852465 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05362418781284854 = (275010.48766881146 - 289757.7017100669) / 275010.48766881146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.4164824270557 > 270.9308018344105.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -84.4011638108043 (T) = (0 -272065.7148001677) / Math.Sqrt((101261.5224972446 / (20)) + (441966.70041789534 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05388187749429565 = (258155.79583456717 - 272065.7148001677) / 258155.79583456717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress(level: Optimal, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 297.7458011287062 > 296.70168234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -148.62331465251418 (T) = (0 -297489.1681221059) / Math.Sqrt((103374.3582741498 / (20)) + (96150.73892757313 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05252066097941168 = (282644.4925511301 - 297489.1681221059) / 282644.4925511301 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Compression.Brotli.Decompress_WithState(level: Fastest, file: "TestDocument.pdf")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 262.27939534722225 > 259.47930234375.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/21/2023 4:02:09 PM, 5/10/2023 2:00:36 PM, 5/16/2023 7:34:13 AM falls between 5/7/2023 5:11:33 AM and 5/16/2023 7:34:13 AM.
IsRegressionStdDev: Marked as regression because -76.42147528289804 (T) = (0 -260977.19148595305) / Math.Sqrt((73102.83454388827 / (20)) + (570478.22836798 / (20))) is less than -2.0243941639098457 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (20) - 2, .025) and -0.05544132342591581 = (247268.30918352964 - 260977.19148595305) / 247268.30918352964 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: DrewScoggins
Labels:

area-System.IO.Compression, os-linux, arch-x64, area-Infrastructure-coreclr, runtime-coreclr, PGO
Milestone: 8.0.0

@agocke
Copy link
Member

agocke commented Sep 5, 2023

If this is just the toolchain update, we don't have any action here. The owners of Brotli could bring it up with the LLVM project, but there's nothing for runtime to do.

@agocke agocke closed this as completed Sep 5, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Oct 6, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@DrewScoggins DrewScoggins

Labels
arch-x64 area-Infrastructure-coreclr os-linux Linux OS (any supported distro) PGO runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
4 participants
@agocke @EgorBo @ViktorHofer @DrewScoggins