Proposal for managing OS and architecture matrix for testing

[richlander]

We are responsible for a large matrix of OS and architecture combinations. Testing all combinations is a waste. Maintaining images for all combinations is equally a waste.

Proposal:

• Test on oldest supported version for x64
• Testing on newest supported version for Arm32 and Arm64

Details

For Ubuntu, we currently support the following versions:

• 18.04
• 20.04
• 22.04
• 24.04

That's also ignoring the two or so non-LTS versions in flight at any one time.

We also support three architectures:

• Arm32
• Arm64
• x64 (AMD64)

That's 12 combinations. However, we typically test just one Ubuntu version for each architecture to keep the matrix more sane.

The same is true for other distros we support (Alpine, Debian, RHEL family, ...).

Assumptions:

• We have plenty of users wanting to run apps on old OSes. The bulk of those folks are on x64.
• Most of the Arm64 usage is on cloud servers, where best-in-class performance and cost is valued, biasing to newer OS versions.
• We reset Arm32 recently to require a recent OS version due to Y2038 support.

Proposal (assuming one distro version per architecture):

• Use x64 to validate the min-supported version
• Use Arm32 and Arm64 to validate the max-supported version

In practice, it is very uncommon to find issues "in the middle". We can rely on customer usage/reports for that.

If we opt to test more than one distro version per architecture (perhaps in outerloop), we should test the opposite of the proposal above for the given architecture.

Let's assume we only test Arm32 on Ubuntu 24.04 (for Ubuntu), then we should delete the Arm32 images for earlier Ubuntu versions and update pipelines to match.

Examples helix image use:

• https://github.com/dotnet/runtime/blob/main/eng/pipelines/libraries/helix-queues-setup.yml
• https://github.com/dotnet/runtime/blob/main/eng/pipelines/coreclr/templates/helix-queues-setup.yml

[richlander]

@sbomer @jkoritzinsky @mthalman @MichaelSimons @jkotas

[jkotas]

We do have different level testing for different areas of the product.

For the runtime tests (that is JIT, GC, loader, interop), we are on the one distro per architecture plan. The exact distro versions do not matter a whole lot. If you would like to tweak the distro versions that we are run the runtime tests on, I do not see a problem with that.

For the libraries tests, we have opted to have broader distro coverage. The libraries tend to be sensitive to distro differences. For example, number of tests have conditions to account for distro differences that we consider acceptable. The adjustment of the matrix for libraries tests should be discussed with and approved by @dotnet/area-infrastructure-libraries.

[MichaelSimons]

Is this policy intended to apply to main as well as the servicing branches? I assume so. I can envision some nuanced limitations that may apply. For example we may choose to not test latest distro version if released in the last n months of support for .NET versions.