Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Linux/x64: 40 Regressions on 5/27/2023 12:14:23 AM #86902

Closed
performanceautofiler bot opened this issue May 30, 2023 · 15 comments
Closed

[Perf] Linux/x64: 40 Regressions on 5/27/2023 12:14:23 AM #86902

performanceautofiler bot opened this issue May 30, 2023 · 15 comments
Assignees
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime
Milestone

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf8 - Duration of single invocation 1.87 ms 2.34 ms 1.25 0.02 False
WriteBasicUtf16 - Duration of single invocation 1.94 ms 2.39 ms 1.23 0.01 False
WriteBasicUtf8 - Duration of single invocation 520.95 ns 600.44 ns 1.15 0.01 False
WriteBasicUtf16 - Duration of single invocation 1.93 ms 2.32 ms 1.20 0.01 False
WriteBasicUtf8 - Duration of single invocation 1.83 ms 2.39 ms 1.30 0.02 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3408806339031343 > 1.92154505949183.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -32.125924874135094 (T) = (0 -2311425.3056228026) / Math.Sqrt((916492552.4091985 / (38)) + (1219546242.0427265 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2651487721284224 = (1826998.8135341406 - 2311425.3056228026) / 1826998.8135341406 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3893149670329668 > 1.997571935011744.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -30.551419155101406 (T) = (0 -2415472.5066380375) / Math.Sqrt((2173440929.3461156 / (38)) + (1427914196.8943276 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.27764027109606826 = (1890573.23981017 - 2415472.5066380375) / 1890573.23981017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 10)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 600.4378005852681 > 550.2634042503744.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -13.555942787766599 (T) = (0 -603.4032247875114) / Math.Sqrt((729.6826798823377 / (38)) + (20.25989565113668 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.11950892195161956 = (538.9892058525174 - 603.4032247875114) / 538.9892058525174 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3185141816239314 > 2.031272294717874.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -29.273890304780583 (T) = (0 -2319651.048315165) / Math.Sqrt((3274466224.0751367 / (38)) + (895885445.9605526 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2401709068259935 = (1870428.5317028742 - 2319651.048315165) / 1870428.5317028742 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3887269498412698 > 1.9251214242859795.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -34.45603805502824 (T) = (0 -2398405.953772479) / Math.Sqrt((1965339907.7037337 / (38)) + (1110426259.2088099 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2838082011740852 = (1868196.4732574986 - 2398405.953772479) / 1868196.4732574986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteAsciiCharArray - Duration of single invocation 363.88 ns 466.47 ns 1.28 0.03 False
WriteAsciiCharArray - Duration of single invocation 105.67 μs 114.49 μs 1.08 0.03 False
WriteAsciiString - Duration of single invocation 363.79 ns 466.84 ns 1.28 0.02 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 466.4734139963222 > 383.15772596179494.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/11/2023 5:02:01 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -62.654031829913194 (T) = (0 -469.92013652447446) / Math.Sqrt((26.245500407837604 / (38)) + (12.25300946969347 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2827261375970982 = (366.3448671941504 - 469.92013652447446) / 366.3448671941504 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.49047945559612 > 113.30206172402872.
IsChangePoint: Marked as a change because one of 4/29/2023 8:32:21 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -13.19588940047768 (T) = (0 -115702.83254668942) / Math.Sqrt((6515911.940266402 / (38)) + (867363.3301534103 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.06850743382361006 = (108284.5368072467 - 115702.83254668942) / 108284.5368072467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 466.84031777918227 > 383.94951849473836.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/11/2023 5:02:01 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -54.041525304572716 (T) = (0 -468.97358170633976) / Math.Sqrt((15.48228807803085 / (38)) + (19.27088055444809 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.28077702098236795 = (366.1633321205534 - 468.97358170633976) / 366.1633321205534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToStream - Duration of single invocation 5.35 μs 6.79 μs 1.27 0.01 False
SerializeObjectProperty - Duration of single invocation 6.23 μs 7.77 μs 1.25 0.01 False
SerializeToString - Duration of single invocation 5.95 μs 7.61 μs 1.28 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 5.58 μs 7.15 μs 1.28 0.01 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToStream(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.785466742077336 > 5.620330566796158.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -38.14413042760439 (T) = (0 -6792.760940070647) / Math.Sqrt((36321.29587460243 / (37)) + (1437.6788092482238 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.24415224434961832 = (5459.750581908542 - 6792.760940070647) / 5459.750581908542 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeObjectProperty(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.770807751279954 > 6.5013251281869975.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -30.080745057111375 (T) = (0 -7658.5811372684) / Math.Sqrt((45506.34930826978 / (37)) + (5038.4359784384305 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.21755943103309572 = (6290.108673192335 - 7658.5811372684) / 6290.108673192335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToString(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.610254915572233 > 6.255681012242412.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -32.68505020007108 (T) = (0 -7491.1123978439255) / Math.Sqrt((32617.196150649073 / (38)) + (6809.848114679876 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.24192928079182446 = (6031.834915002383 - 7491.1123978439255) / 6031.834915002383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.151032749657221 > 5.859381495053741.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -35.90800319403818 (T) = (0 -7121.446313133833) / Math.Sqrt((29809.964933499366 / (38)) + (4944.305579525699 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.25348767610064143 = (5681.305408033432 - 7121.446313133833) / 5681.305408033432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
StaticMethod4_int_string_struct_class - Duration of single invocation 49.76 ns 52.27 ns 1.05 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Invoke.StaticMethod4_int_string_struct_class


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.266650549909954 > 50.215607555651395.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -17.12534861116317 (T) = (0 -52.382046518807556) / Math.Sqrt((1.4405795219331046 / (38)) + (0.04899866671257257 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.07547379950348426 = (48.70601826189616 - 52.382046518807556) / 48.70601826189616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InterpolateIntoSpan_NonFlags - Duration of single invocation 72.38 ns 87.61 ns 1.21 0.06 False
InterpolateIntoSpan_Flags - Duration of single invocation 75.00 ns 83.24 ns 1.11 0.04 False
InterpolateIntoSpan_Flags - Duration of single invocation 64.75 ns 71.04 ns 1.10 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 87.6100956968936 > 76.50684204353036.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/21/2023 1:46:51 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -7.481037391105198 (T) = (0 -85.35887906495647) / Math.Sqrt((11.593569699738323 / (37)) + (7.602943293148558 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.12382509730703668 = (75.95388220951595 - 85.35887906495647) / 75.95388220951595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: 32)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.24290799504712 > 81.17197146223633.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -11.57857230318279 (T) = (0 -83.14123999437209) / Math.Sqrt((10.56402685678745 / (38)) + (0.907792668164346 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.10040904835775401 = (75.55484946116331 - 83.14123999437209) / 75.55484946116331 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: Red)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 71.03822988869678 > 68.96792473499272.
IsChangePoint: Marked as a change because one of 3/22/2023 1:04:34 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -11.429818712205602 (T) = (0 -72.97867041955617) / Math.Sqrt((3.5585694283905367 / (38)) + (2.4303351757881004 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.1243563971602053 = (64.90706203467059 - 72.97867041955617) / 64.90706203467059 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EscapeDataString - Duration of single invocation 6.16 μs 12.80 μs 2.08 0.05 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.800517183048434 > 6.483500356773706.
IsChangePoint: Marked as a change because one of 5/10/2023 6:05:58 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -99.0120814232714 (T) = (0 -12786.393197676012) / Math.Sqrt((177525.3632517209 / (37)) + (124.92219905417701 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -1.1623430475639271 = (5913.212157562616 - 12786.393197676012) / 5913.212157562616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels May 30, 2023
@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOf_Word_NotFound - Duration of single invocation 405.85 ns 557.46 ns 1.37 0.09 False
IndexOf_Word_NotFound - Duration of single invocation 411.04 ns 557.30 ns 1.36 0.09 False
LastIndexOf_Word_NotFound - Duration of single invocation 416.49 ns 690.26 ns 1.66 0.16 False
IndexOf_Word_NotFound - Duration of single invocation 406.14 ns 557.95 ns 1.37 0.07 False
LastIndexOf_Word_NotFound - Duration of single invocation 377.18 ns 550.69 ns 1.46 0.07 False
LastIndexOf_Word_NotFound - Duration of single invocation 418.49 ns 792.76 ns 1.89 0.15 False

graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, None, False))


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 557.4559374472741 > 425.6652644212752.
IsChangePoint: Marked as a change because one of 3/8/2023 10:34:21 AM, 4/25/2023 3:46:39 AM, 5/15/2023 9:18:21 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -125.78287533083699 (T) = (0 -557.2328841999168) / Math.Sqrt((16.217599109100632 / (38)) + (5.886258179278807 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.36580236347710837 = (407.9893981009767 - 557.2328841999168) / 407.9893981009767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, None, False))


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 557.3047942305857 > 425.80089114613673.
IsChangePoint: Marked as a change because one of 3/8/2023 10:34:21 AM, 4/25/2023 3:46:39 AM, 5/15/2023 9:18:21 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -144.88652499607846 (T) = (0 -557.0576792741614) / Math.Sqrt((18.416552200893843 / (38)) + (3.366451790019962 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.3623457498743924 = (408.89596442424534 - 557.0576792741614) / 408.89596442424534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, IgnoreCase, False))


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 690.2581159493576 > 438.0207919511483.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/15/2023 9:18:21 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -13.89956925110726 (T) = (0 -725.9599504121796) / Math.Sqrt((279.7474848662933 / (38)) + (2761.6256766343045 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.706597872862117 = (425.3843052052327 - 725.9599504121796) / 425.3843052052327 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreNonSpace, False))


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 557.9476998326733 > 425.1856524573894.
IsChangePoint: Marked as a change because one of 3/8/2023 10:34:21 AM, 4/25/2023 3:46:39 AM, 5/15/2023 9:18:21 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -114.16025287394048 (T) = (0 -556.5581935876322) / Math.Sqrt((21.003758835223852 / (38)) + (6.7932698718628215 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.3628702329182873 = (408.37211067108353 - 556.5581935876322) / 408.37211067108353 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (, None, False))


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 550.6915677721328 > 387.2967680124046.
IsChangePoint: Marked as a change because one of 3/8/2023 10:34:21 AM, 4/25/2023 3:46:39 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -4.295362512344054 (T) = (0 -503.3537298282795) / Math.Sqrt((319.73256800566577 / (38)) + (5503.7913713100825 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.350686636382167 = (372.665069950288 - 503.3537298282795) / 372.665069950288 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringSearch.LastIndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 792.755511025433 > 437.2767627338758.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/15/2023 9:18:21 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -20.228937274823224 (T) = (0 -775.7221820588065) / Math.Sqrt((272.6929682350362 / (38)) + (1733.6316487226147 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.8140331421669419 = (427.6229380970253 - 775.7221820588065) / 427.6229380970253 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.Perf_StreamWriter

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteCharArray - Duration of single invocation 294.25 ms 322.09 ms 1.09 0.06 False
WriteFormat - Duration of single invocation 39.06 ns 41.24 ns 1.06 0.16 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_StreamWriter*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_StreamWriter.WriteCharArray(writeLength: 100)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 322.08740133333333 > 307.4210430625.
IsChangePoint: Marked as a change because one of 3/17/2023 1:30:52 PM, 5/5/2023 3:18:41 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -19.609912598812798 (T) = (0 -322333485.5222222) / Math.Sqrt((47720419999069.09 / (38)) + (403477287864.476 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.07524269230342202 = (299777424.975294 - 322333485.5222222) / 299777424.975294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.Perf_StreamWriter.WriteFormat


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.23892068716091 > 40.294353558993606.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -7.175506109206024 (T) = (0 -42.84767618404021) / Math.Sqrt((4.05025766814677 / (37)) + (2.047132336408836 / (7))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (7) - 2, .025) and -0.11877790450680994 = (38.29864355689856 - 42.84767618404021) / 38.29864355689856 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.MDBenchF.MDInvMt

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 2.37 ms 3.03 ms 1.28 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchF.MDInvMt*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchF.MDInvMt.Test


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.033135774583333 > 2.4932704231009617.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -58.17881435490348 (T) = (0 -3035092.5423336383) / Math.Sqrt((4375129664.734865 / (38)) + (18728359.087703746 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2633466208745493 = (2402422.654388074 - 3035092.5423336383) / 2402422.654388074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseUInt64Hex - Duration of single invocation 4.82 ns 5.83 ns 1.21 0.26 False
TryParseUInt64Hex - Duration of single invocation 11.29 ns 19.02 ns 1.68 0.23 False
TryParseUInt64 - Duration of single invocation 15.70 ns 21.82 ns 1.39 0.02 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64Hex(value: 3039)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.825899528304139 > 5.064194278945534.
IsChangePoint: Marked as a change because one of 4/17/2023 6:40:18 PM, 5/2/2023 4:25:00 AM, 5/11/2023 1:19:27 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -25.91343300307834 (T) = (0 -5.825409787829563) / Math.Sqrt((0.062952525096413 / (38)) + (7.794340251816787E-05 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.22214222377075307 = (4.766556358601257 - 5.825409787829563) / 4.766556358601257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64Hex(value: FFFFFFFFFFFFFFFF)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.015682261431582 > 11.853379952053345.
IsChangePoint: Marked as a change because one of 4/17/2023 11:29:15 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -70.79208609612687 (T) = (0 -18.79551706896488) / Math.Sqrt((0.2086071827224402 / (38)) + (0.037371523284709066 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.6883900774475545 = (11.132212466789218 - 18.79551706896488) / 11.132212466789218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Buffers.Text.Tests.Utf8ParserTests.TryParseUInt64(value: 18446744073709551615)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.82281660426052 > 16.48157037091468.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -26.992921617292286 (T) = (0 -22.09044533223275) / Math.Sqrt((1.2367098928688385 / (38)) + (0.07538638323802585 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.3504841822623753 = (16.357426190084 - 22.09044533223275) / 16.357426190084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<IntStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqQuery - Duration of single invocation 18.64 μs 21.06 μs 1.13 0.06 False
LinqOrderByExtension - Duration of single invocation 15.91 μs 21.67 μs 1.36 0.04 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;IntStruct&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<IntStruct>.LinqQuery(Size: 512)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.06216444 > 16.708420036.
IsChangePoint: Marked as a change because one of 3/25/2023 10:49:06 AM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -3.336414956339992 (T) = (0 -22839.617866190474) / Math.Sqrt((16286447.531842625 / (38)) + (5175294.03850908 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.19902415076444238 = (19048.505279588397 - 22839.617866190474) / 19048.505279588397 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<IntStruct>.LinqOrderByExtension(Size: 512)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.666328071428573 > 16.33941273.
IsChangePoint: Marked as a change because one of 3/25/2023 10:49:06 AM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -4.02643711215368 (T) = (0 -22852.181161111108) / Math.Sqrt((16216120.398668656 / (37)) + (3115812.0209701895 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.20833784006054434 = (18912.07938995446 - 22852.181161111108) / 18912.07938995446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Guid

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 20.96 ns 25.40 ns 1.21 0.01 False
ParseExactD - Duration of single invocation 23.53 ns 26.01 ns 1.11 0.03 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Guid*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Guid.Parse


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.399690132915698 > 22.02469566640517.
IsChangePoint: Marked as a change because one of 4/1/2023 1:31:30 PM, 4/25/2023 6:46:56 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -69.43625861203486 (T) = (0 -25.41973502576306) / Math.Sqrt((0.12077176086103693 / (38)) + (0.003171535351769791 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.19948448377276554 = (21.19221663110622 - 25.41973502576306) / 21.19221663110622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Guid.ParseExactD


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.010506410635752 > 24.510188216447286.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -31.86435954052116 (T) = (0 -25.833578849462892) / Math.Sqrt((0.038973121822451204 / (38)) + (0.028556475244863443 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.10352763741784099 = (23.409997152324365 - 25.833578849462892) / 23.409997152324365 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqOrderByExtension - Duration of single invocation 19.65 μs 24.65 μs 1.25 0.37 False
LinqQuery - Duration of single invocation 19.60 μs 24.62 μs 1.26 0.43 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<Int32>.LinqOrderByExtension(Size: 512)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.645369799999997 > 20.961033819500003.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -7.995370757878904 (T) = (0 -24454.909150000003) / Math.Sqrt((16178656.642191786 / (38)) + (35004.39025593646 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2735392586936132 = (19202.320606186622 - 24454.909150000003) / 19202.320606186622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<Int32>.LinqQuery(Size: 512)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.624822653846152 > 20.70618416596154.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -6.009450973642971 (T) = (0 -24245.24607527473) / Math.Sqrt((15903970.829460373 / (38)) + (993261.6889140218 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.23369415218488124 = (19652.558158224405 - 24245.24607527473) / 19652.558158224405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 19.83 ns 25.89 ns 1.31 0.02 False
IndexOfString - Duration of single invocation 27.99 ns 36.97 ns 1.32 0.02 False
IndexOfString - Duration of single invocation 17.43 ns 23.04 ns 1.32 0.03 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.89154351220897 > 20.756261680525736.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -9.921972651490789 (T) = (0 -25.60540846575276) / Math.Sqrt((8.159092419223517 / (38)) + (0.052285725921656125 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.22423232890127828 = (20.91548136842053 - 25.60540846575276) / 20.91548136842053 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.973547290262744 > 29.381180263733057.
IsChangePoint: Marked as a change because one of 3/8/2023 10:34:21 AM, 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -17.4775032902635 (T) = (0 -36.83105114392996) / Math.Sqrt((7.558831078081545 / (38)) + (0.02248832840150166 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.27165968801591933 = (28.962977666921915 - 36.83105114392996) / 28.962977666921915 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.042912979137256 > 18.30144091040031.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -8.794180629300222 (T) = (0 -23.228700158103738) / Math.Sqrt((8.741782720318247 / (38)) + (0.18867480004826662 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2400758803779626 = (18.73167644468971 - 23.228700158103738) / 18.73167644468971 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<BigStruct>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LinqOrderByExtension - Duration of single invocation 18.24 μs 31.04 μs 1.70 0.16 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;BigStruct&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<BigStruct>.LinqOrderByExtension(Size: 512)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.044924885714277 > 19.395351039076925.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -5.3013711849940455 (T) = (0 -29191.9249660928) / Math.Sqrt((37703307.309724994 / (38)) + (8348933.238877036 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.3896254556291507 = (21007.045350127242 - 29191.9249660928) / 21007.045350127242 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.MDBenchI.MDMulMatrix

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 682.00 ms 849.36 ms 1.25 0.01 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDMulMatrix*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDMulMatrix.Test


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 849.3581336 > 720.190938565.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -12.035769568749958 (T) = (0 -853158295.12558) / Math.Sqrt((4131815360925108.5 / (37)) + (149317695020701.56 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.19739634523559418 = (712511190.2339375 - 853158295.12558) / 712511190.2339375 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@cincuranet
Copy link
Contributor

The diff is "always" the same: 2e45de5...b2edecd.

@cincuranet cincuranet removed the untriaged New issue has not been triaged by the area owner label May 30, 2023
@ghost ghost added the untriaged New issue has not been triaged by the area owner label May 30, 2023
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 30, 2023
@cincuranet cincuranet transferred this issue from dotnet/perf-autofiling-issues May 30, 2023
@kunalspathak
Copy link
Member

@jakobbotsch - could it be from one of your physical promotion changes? Remind me, but it is not yet on by default, yes?

@kunalspathak kunalspathak added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label May 30, 2023
@ghost
Copy link

ghost commented May 30, 2023

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf8 - Duration of single invocation 1.87 ms 2.34 ms 1.25 0.02 False
WriteBasicUtf16 - Duration of single invocation 1.94 ms 2.39 ms 1.23 0.01 False
WriteBasicUtf8 - Duration of single invocation 520.95 ns 600.44 ns 1.15 0.01 False
WriteBasicUtf16 - Duration of single invocation 1.93 ms 2.32 ms 1.20 0.01 False
WriteBasicUtf8 - Duration of single invocation 1.83 ms 2.39 ms 1.30 0.02 False

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3408806339031343 > 1.92154505949183.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -32.125924874135094 (T) = (0 -2311425.3056228026) / Math.Sqrt((916492552.4091985 / (38)) + (1219546242.0427265 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2651487721284224 = (1826998.8135341406 - 2311425.3056228026) / 1826998.8135341406 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3893149670329668 > 1.997571935011744.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -30.551419155101406 (T) = (0 -2415472.5066380375) / Math.Sqrt((2173440929.3461156 / (38)) + (1427914196.8943276 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.27764027109606826 = (1890573.23981017 - 2415472.5066380375) / 1890573.23981017 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 10)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 600.4378005852681 > 550.2634042503744.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -13.555942787766599 (T) = (0 -603.4032247875114) / Math.Sqrt((729.6826798823377 / (38)) + (20.25989565113668 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.11950892195161956 = (538.9892058525174 - 603.4032247875114) / 538.9892058525174 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3185141816239314 > 2.031272294717874.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -29.273890304780583 (T) = (0 -2319651.048315165) / Math.Sqrt((3274466224.0751367 / (38)) + (895885445.9605526 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2401709068259935 = (1870428.5317028742 - 2319651.048315165) / 1870428.5317028742 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3887269498412698 > 1.9251214242859795.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/25/2023 12:29:35 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -34.45603805502824 (T) = (0 -2398405.953772479) / Math.Sqrt((1965339907.7037337 / (38)) + (1110426259.2088099 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2838082011740852 = (1868196.4732574986 - 2398405.953772479) / 1868196.4732574986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteAsciiCharArray - Duration of single invocation 363.88 ns 466.47 ns 1.28 0.03 False
WriteAsciiCharArray - Duration of single invocation 105.67 μs 114.49 μs 1.08 0.03 False
WriteAsciiString - Duration of single invocation 363.79 ns 466.84 ns 1.28 0.02 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 8000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 466.4734139963222 > 383.15772596179494.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/11/2023 5:02:01 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -62.654031829913194 (T) = (0 -469.92013652447446) / Math.Sqrt((26.245500407837604 / (38)) + (12.25300946969347 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.2827261375970982 = (366.3448671941504 - 469.92013652447446) / 366.3448671941504 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiCharArray(StringLengthInChars: 2000000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 114.49047945559612 > 113.30206172402872.
IsChangePoint: Marked as a change because one of 4/29/2023 8:32:21 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -13.19588940047768 (T) = (0 -115702.83254668942) / Math.Sqrt((6515911.940266402 / (38)) + (867363.3301534103 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.06850743382361006 = (108284.5368072467 - 115702.83254668942) / 108284.5368072467 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 8000)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 466.84031777918227 > 383.94951849473836.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/11/2023 5:02:01 PM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -54.041525304572716 (T) = (0 -468.97358170633976) / Math.Sqrt((15.48228807803085 / (38)) + (19.27088055444809 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.28077702098236795 = (366.1633321205534 - 468.97358170633976) / 366.1633321205534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToStream - Duration of single invocation 5.35 μs 6.79 μs 1.27 0.01 False
SerializeObjectProperty - Duration of single invocation 6.23 μs 7.77 μs 1.25 0.01 False
SerializeToString - Duration of single invocation 5.95 μs 7.61 μs 1.28 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 5.58 μs 7.15 μs 1.28 0.01 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToStream(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.785466742077336 > 5.620330566796158.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -38.14413042760439 (T) = (0 -6792.760940070647) / Math.Sqrt((36321.29587460243 / (37)) + (1437.6788092482238 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.24415224434961832 = (5459.750581908542 - 6792.760940070647) / 5459.750581908542 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeObjectProperty(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.770807751279954 > 6.5013251281869975.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -30.080745057111375 (T) = (0 -7658.5811372684) / Math.Sqrt((45506.34930826978 / (37)) + (5038.4359784384305 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.21755943103309572 = (6290.108673192335 - 7658.5811372684) / 6290.108673192335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToString(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.610254915572233 > 6.255681012242412.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -32.68505020007108 (T) = (0 -7491.1123978439255) / Math.Sqrt((32617.196150649073 / (38)) + (6809.848114679876 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.24192928079182446 = (6031.834915002383 - 7491.1123978439255) / 6031.834915002383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Text.Json.Serialization.Tests.WriteJson<ImmutableSortedDictionary<String, String>>.SerializeToUtf8Bytes(Mode: SourceGen)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.151032749657221 > 5.859381495053741.
IsChangePoint: Marked as a change because one of 5/5/2023 3:18:41 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -35.90800319403818 (T) = (0 -7121.446313133833) / Math.Sqrt((29809.964933499366 / (38)) + (4944.305579525699 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.25348767610064143 = (5681.305408033432 - 7121.446313133833) / 5681.305408033432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
StaticMethod4_int_string_struct_class - Duration of single invocation 49.76 ns 52.27 ns 1.05 0.09 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Invoke.StaticMethod4_int_string_struct_class


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 52.266650549909954 > 50.215607555651395.
IsChangePoint: Marked as a change because one of 4/12/2023 9:47:42 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -17.12534861116317 (T) = (0 -52.382046518807556) / Math.Sqrt((1.4405795219331046 / (38)) + (0.04899866671257257 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.07547379950348426 = (48.70601826189616 - 52.382046518807556) / 48.70601826189616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InterpolateIntoSpan_NonFlags - Duration of single invocation 72.38 ns 87.61 ns 1.21 0.06 False
InterpolateIntoSpan_Flags - Duration of single invocation 75.00 ns 83.24 ns 1.11 0.04 False
InterpolateIntoSpan_Flags - Duration of single invocation 64.75 ns 71.04 ns 1.10 0.07 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 87.6100956968936 > 76.50684204353036.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/21/2023 1:46:51 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -7.481037391105198 (T) = (0 -85.35887906495647) / Math.Sqrt((11.593569699738323 / (37)) + (7.602943293148558 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -0.12382509730703668 = (75.95388220951595 - 85.35887906495647) / 75.95388220951595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: 32)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.24290799504712 > 81.17197146223633.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -11.57857230318279 (T) = (0 -83.14123999437209) / Math.Sqrt((10.56402685678745 / (38)) + (0.907792668164346 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.10040904835775401 = (75.55484946116331 - 83.14123999437209) / 75.55484946116331 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: Red)


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 71.03822988869678 > 68.96792473499272.
IsChangePoint: Marked as a change because one of 3/22/2023 1:04:34 PM, 5/19/2023 10:30:38 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -11.429818712205602 (T) = (0 -72.97867041955617) / Math.Sqrt((3.5585694283905367 / (38)) + (2.4303351757881004 / (6))) is less than -2.0180817028167235 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (38) + (6) - 2, .025) and -0.1243563971602053 = (64.90706203467059 - 72.97867041955617) / 64.90706203467059 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 18.04
Queue TigerUbuntu
Baseline 7b2dc26555db2ef75c4ef822613a63b8c1d25e85
Compare 71738239969b184b66e72a927a527690a761ecbf
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Uri

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EscapeDataString - Duration of single invocation 6.16 μs 12.80 μs 2.08 0.05 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Uri*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Uri.EscapeDataString(input: "{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{{")


Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.800517183048434 > 6.483500356773706.
IsChangePoint: Marked as a change because one of 5/10/2023 6:05:58 AM, 5/26/2023 5:44:43 PM, 5/29/2023 1:48:21 PM falls between 5/21/2023 1:46:51 AM and 5/29/2023 1:48:21 PM.
IsRegressionStdDev: Marked as regression because -99.0120814232714 (T) = (0 -12786.393197676012) / Math.Sqrt((177525.3632517209 / (37)) + (124.92219905417701 / (6))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (6) - 2, .025) and -1.1623430475639271 = (5913.212157562616 - 12786.393197676012) / 5913.212157562616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

os-linux, arch-x64, area-CodeGen-coreclr, untriaged, runtime-coreclr, needs-area-label

Milestone: -

@cincuranet
Copy link
Contributor

@jakobbotsch
Copy link
Member

@jakobbotsch - could it be from one of your physical promotion changes? Remind me, but it is not yet on by default, yes?

Correct, not enabled yet. In theory it could be #86774, but that had quite few affected contexts in benchmarks. What about #86551 @EgorBo?

@AndyAyersMS
Copy link
Member

@EgorBo's CI run reported minimal diffs but suspect this was misleading as there were quite a high number of missed contexts:

image

@EgorBo
Copy link
Member

EgorBo commented May 31, 2023

@EgorBo's CI run reported minimal diffs but suspect this was misleading as there were quite a high number of missed contexts:

I used a trick to avoid introducing missing contexts (see #86551 (comment)) and didn't see any diffs (as expected). I'll double check

@EgorBo
Copy link
Member

EgorBo commented May 31, 2023

Ok, I think I know what it is - it's indeed could be #86551 namely, the last minute "cosmetic" change in impMarkInlineCandidateHelper dd79675 - the destructor of InlineResult calls vm and tells it "this method was not profitable to inline - mark it as noinline for future compilations"

and I broke it. So regressions come from excessive inlining/"run out of budget", e.g. I checked Guid.Parse.

This trick we do for TP actually makes little sense since profitability heavily depends on call-site and PGO data, I already know a couple of places where we suffer from it because JIT bakes noinline too early (e.g. some non-hot callsite with non-constant args) and then even constant args don't help to inline a callee which is already has an artififcial noinline on it. I don't think it saves a lot of TP since we have other "early-out" checks like IL size, number of basic-blocks, etc. But we run into the "run out of budget" problem.

Before I "fix" that part, let's wait for tomorrow's perf reports. So far, we see more improvements than regressions from that change, e.g.:

Improvements:

Regressions:

cc @AndyAyersMS since we recently discussed the "noinline" problem in Discord

@AndyAyersMS
Copy link
Member

So far, we see more improvements than regressions from that change,

I'm not sure I completely follow -- you are saying we now mark more methods as noinline, or fewer methods, or just different methods?

@EgorBo
Copy link
Member

EgorBo commented May 31, 2023

So far, we see more improvements than regressions from that change,

I'm not sure I completely follow -- you are saying we now mark more methods as noinline, or fewer methods, or just different methods?

We now mark less methods as noinline in JIT (if mark any at all) based on previous inline decisions.

@EgorBo
Copy link
Member

EgorBo commented Jun 1, 2023

--

@JulieLeeMSFT JulieLeeMSFT removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 2, 2023
@JulieLeeMSFT JulieLeeMSFT added this to the 8.0.0 milestone Jun 2, 2023
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Jun 2, 2023
@EgorBo
Copy link
Member

EgorBo commented Jun 10, 2023

Fixed by #87115

@EgorBo
Copy link
Member

EgorBo commented Jun 11, 2023

All regressions are gone with #87115 as expected

image

image

image

@EgorBo EgorBo closed this as completed Jun 11, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Jul 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime
Projects
None yet
Development

No branches or pull requests

6 participants