Speed up .NET installation process

[maxim-lobanov]

We have investigated and found that installation of .NET is hugest bottleneck in image-generation of Windows / Ubuntu:

• Ubuntu dotnetcore-sdk.sh takes 1h40m (the total time of image-generation is 3h20m)
• Windows Install-DotnetSDK.ps1 takes 1h8m (the total time of image-generation is 6h)

There are two reasons for it:

1. We install a lot of .NET versions: 2.1, 3.1, 5.0 on Windows and 2.1, 3.0, 3.1, 5.0 on Ubuntu
2. We run a huge validation after installation that takes the most of the time:

• On Ubuntu, we run the following code for every version:

    dotnet new globaljson --sdk-version "$sdk"
    dotnet new "$sample"
    dotnet restore
    dotnet build

• On Windows, we run the following code for every version:

    & $env:ProgramFiles\dotnet\dotnet.exe new globaljson --sdk-version "$sdkVersion"
    & $env:ProgramFiles\dotnet\dotnet.exe new $template

Timelines for comparison:

Platform	Downloading and installation time	Validation time
Ubuntu	24 minutes	1h 16 minutes
Windows	14 minutes	54 minutes

[maxim-lobanov]

Proposals

Define timeline for deprecation of .NET 3.0 on Windows images

.NET 3.0 EOL was on March 2020
Our usual approach is deprecating tools in 6 months after they have reached end-of-life so it is a time for .NET 3.0. I think we need to do proper announcement and give customers time to fix pipelines (add UseDotNet task).

Get rid of current validation approach or rework it somehow

We can't afford spending 1h 16 minutes for validation of .NET during image-generation. It is really huge amount of time that is 30% of whole Ubuntu image-generation. it brings additional cost for building self-hosted machines.
Also I have never seen builds where installation of .NET fails on this check.
We suggest:

• reworking current validation approach during image-generation to something simple like dotnet --version for every version like we do for other software tools.
• Move current validation approach to buildcanary. It will also allow us to parallel it to 3-4 machines and reduce to 20 minutes.

[maxim-lobanov]

cc: @alepauly @bishal-pdMSFT

[al-cheb]

cc: @janvorli

[bishal-pdMSFT]

@maxim-lobanov lets talk about two parts:

1. We do indeed install a lot of versions - in-fact we install every available patch version for every release of 2.1, 3.1 and 5.0. We can definitely try to see if we can only install latest feature versions for every release. [Please note that there can be breaking changes across feature versions and hence we might want to install such feature versions.]. One problem we will face if we only install selected version is that users will have to use UseDotNet task specify a specific version there. And a specific version brings problem of pipeline being stale once that version is stale. This means that user will need to do more maintenance work regularly. UseDotNet task does support global.json, but I am not sure how many users use that feature. May be we can look at telemetry of how many tasks runs use global.json and their success rate.
2. About the validation part. they are not really validation in a strict sense. Ofcourse they provide good validation, but the main reason for running those commands is to warm up dotnet's cache with various package. This speeds up user's pipeline build time and is a very good value add IMO.

[maxim-lobanov]

@catthehacker , that is good idea to parallel validation by build type: "console","mstest","xunit","web","mvc","webapi".
We have tried to parallel by versions but got a lot of issues with locking processes. Your approach should work.
We will try it, thank you!

[catthehacker]

Hi @maxim-lobanov 👋 Were you able to test if performance has improved/degraded and how noticeable it is? I've also replaced $ThrottleLimit = nproc with $ThrottleLimit = [System.Environment]::ProcessorCount in script above.
If you would like, I could make a PR for it. 🐱‍💻

[maxim-lobanov]

Unfortunately, not yet. I will look closer next week.
Based on my previous investigation, the warm up process doesn't make much sense because during warm up, dotnet download and cache NuGet packages to put it under current user directory. During image deployment (Hosted images) we re-create user and the nuget cache is reset. I need additional time to double check and confirm

[catthehacker]

Looked it up a bit

• option 1: Use env var NUGET_PACKAGES to specify different location (source: https://docs.microsoft.com/en-us/nuget/consume-packages/managing-the-global-packages-and-cache-folders)
• option 2:
  Perhaps something like this would work:

New-Item -Path /usr/share -Name nuget -ItemType Directory
New-Item -Path /etc/skel -Name .nuget -ItemType SymbolicLink -Value /usr/share/nuget
New-Item -Path $env:HOME -Name .nuget -ItemType SymbolicLink -Value /usr/share/nuget 

mkdir -p /usr/share/nuget
ln -s /usr/share/nuget $HOME/.nuget
ln -s /usr/share/nuget /etc/skel/.nuget

Or it could do both

[catthehacker]

and it seems that .dotnet in user directory is also gone

/home/runner:
total 60K
drwxrwxrwx  11 runner docker 4.0K Feb  1 00:56 .
drwxr-xr-x+  5 root   root   4.0K Jan 28 16:42 ..
-rwxrwxrwx   1 runner docker  220 Jan 28 16:42 .bash_logout
-rwxrwxrwx   1 runner docker   93 Jan 28 16:42 .bash_profile
-rwxrwxrwx   1 runner docker 3.9K Jan 28 16:42 .bashrc
lrwxrwxrwx   1 runner docker   22 Jan 28 16:42 .cargo -> /usr/share/rust/.cargo
drwxrwxrwx   2 runner docker 4.0K Jan 28 16:42 .composer
drwxrwxrwx   4 runner docker 4.0K Feb  1 00:56 .config
drwx------   2 runner docker 4.0K Jan 28 16:46 .docker
drwxrwxrwx   6 runner docker 4.0K Jan 28 16:42 .nvm
-rwxrwxrwx   1 runner docker  807 Jan 28 16:42 .profile
lrwxrwxrwx   1 runner docker   23 Jan 28 16:42 .rustup -> /usr/share/rust/.rustup
drwxr-xr-x   2 runner root   4.0K Jan 28 16:47 factory
drwxr-xr-x   2 runner docker 4.0K Feb  1 00:56 perflog
drwxr-xr-x   3 runner root   4.0K Jan 28 16:46 runners
drwxr-xr-x   2 runner root   4.0K Jan 28 16:46 warmup
drwxr-xr-x   6 runner root   4.0K Feb  1 00:56 work

[bishal-pdMSFT]

/cc @tejasd1990

[al-cheb]

1. Windows performance tests

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	console	00:01:23.1422020	00:06:29.6115216	-00:05:06.4693196
2.1.403	console	00:01:06.4617772	00:06:56.7557985	-00:05:50.2940213
2.1.522	console	00:01:27.3826393	00:07:02.5500234	-00:05:35.1673841
2.1.617	console	00:01:23.6427146	00:06:55.0966238	-00:05:31.4539092
2.1.701	console	00:01:25.0246426	00:06:02.6054981	-00:04:37.5808555
2.1.814	console	00:01:24.3710427	00:08:02.4877003	-00:06:38.1166576
3.1.113	console	00:00:58.4776229	00:01:11.2263500	-00:00:12.7487271
3.1.202	console	00:01:29.0254563	00:01:12.3604219	00:00:16.6650344
3.1.302	console	00:00:56.4902593	00:01:11.6962947	-00:00:15.2060354
3.1.407	console	00:01:29.0882603	00:01:11.1680067	00:00:17.9202536
5.0.104	console	00:00:47.5797408	00:00:29.2596272	00:00:18.3201136
5.0.201	console	00:00:52.5773291	00:00:46.1813610	00:00:06.3959681

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	mstest	00:02:43.5928440	00:03:55.9030091	-00:01:12.3101651
2.1.403	mstest	00:00:47.9393689	00:03:57.5678919	-00:03:09.6285230
2.1.522	mstest	00:00:48.3827603	00:04:12.3853303	-00:03:24.0025700
2.1.617	mstest	00:00:47.6228671	00:03:40.1556762	-00:02:52.5328091
2.1.701	mstest	00:01:56.6080163	00:03:38.1704822	-00:01:41.5624659
2.1.814	mstest	00:01:59.7814258	00:03:54.3917846	-00:01:54.6103588
3.1.113	mstest	00:02:42.4821519	00:01:35.9432807	00:01:06.5388712
3.1.202	mstest	00:00:47.4475314	00:01:48.6269294	-00:01:01.1793980
3.1.302	mstest	00:00:48.7652955	00:01:39.2030542	-00:00:50.4377587
3.1.407	mstest	00:02:42.1605420	00:01:36.4117498	00:01:05.7487922
5.0.104	mstest	00:01:57.6444603	00:01:42.8538962	00:00:14.7905641
5.0.201	mstest	00:00:41.7246469	00:01:44.9193834	-00:01:03.1947365

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	web	00:02:35.2181383	00:03:28.4236830	-00:00:53.2055447
2.1.403	web	00:02:36.2708485	00:03:53.1205567	-00:01:16.8497082
2.1.522	web	00:02:36.3495039	00:04:09.0655448	-00:01:32.7160409
2.1.617	web	00:01:01.2713103	00:03:39.3769622	-00:02:38.1056519
2.1.701	web	00:02:34.1771345	00:03:51.9176364	-00:01:17.7405019
2.1.814	web	00:02:34.2907009	00:03:39.6337280	-00:01:05.3430271
3.1.113	web	00:01:08.0777820	00:00:45.6366197	00:00:22.4411623
3.1.202	web	00:01:09.8168958	00:00:46.6964303	00:00:23.1204655
3.1.302	web	00:00:33.4464059	00:00:45.1800563	-00:00:11.7336504
3.1.407	web	00:01:04.9385268	00:00:44.0782681	00:00:20.8602587
5.0.104	web	00:00:58.4954738	00:00:33.3440077	00:00:25.1514661
5.0.201	web	00:00:18.4001495	00:00:28.1330039	-00:00:09.7328544

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	mvc	00:01:00.6451871	00:04:17.4756306	-00:03:16.8304435
2.1.403	mvc	00:01:50.3563926	00:04:21.3662177	-00:02:31.0098251
2.1.522	mvc	00:01:52.6776312	00:05:06.6917469	-00:03:14.0141157
2.1.617	mvc	00:01:51.8431576	00:04:28.9352754	-00:02:37.0921178
2.1.701	mvc	00:01:52.0487230	00:04:15.7640287	-00:02:23.7153057
2.1.814	mvc	00:01:50.0906782	00:04:25.2417494	-00:02:35.1510712
3.1.113	mvc	00:00:51.8725479	00:00:51.5377491	00:00:00.3347988
3.1.202	mvc	00:00:29.2684305	00:00:50.9359259	-00:00:21.6674954
3.1.302	mvc	00:00:27.9950596	00:00:48.5224146	-00:00:20.5273550
3.1.407	mvc	00:00:26.8471322	00:01:07.0562206	-00:00:40.2090884
5.0.104	mvc	00:00:24.2437739	00:00:31.6252562	-00:00:07.3814823
5.0.201	mvc	00:00:17.2093787	00:00:31.3806633	-00:00:14.1712846

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	webapi	00:00:54.9369518	00:02:39.3273228	-00:01:44.3903710
2.1.403	webapi	00:00:58.4424877	00:04:08.6165907	-00:03:10.1741030
2.1.522	webapi	00:01:00.4426711	00:03:59.8014270	-00:02:59.3587559
2.1.617	webapi	00:00:55.6702610	00:04:07.8563726	-00:03:12.1861116
2.1.701	webapi	00:00:52.8854389	00:03:45.0108806	-00:02:52.1254417
2.1.814	webapi	00:01:41.2129295	00:04:18.2443408	-00:02:37.0314113
3.1.113	webapi	00:00:32.7883090	00:00:54.0075181	-00:00:21.2192091
3.1.202	webapi	00:00:29.8772473	00:00:32.1993721	-00:00:02.3221248
3.1.302	webapi	00:00:31.5802219	00:00:42.6361010	-00:00:11.0558791
3.1.407	webapi	00:00:55.5681695	00:00:42.7941136	00:00:12.7740559
5.0.104	webapi	00:00:30.9488312	00:00:38.8750303	-00:00:07.9261991
5.0.201	webapi	00:00:25.3574872	00:00:30.4165166	-00:00:05.0590294

2. Ubuntu performance tests

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	console	00:00:29	00:00:25	-00:00:04
2.1.403	console	00:00:24	00:00:30	00:00:06
2.1.522	console	00:00:20	00:00:31	00:00:11
2.1.617	console	00:00:23	00:00:31	00:00:08
2.1.701	console	00:00:26	00:00:30	00:00:04
2.1.814	console	00:00:33	00:00:31	-00:00:02
3.1.113	console	00:00:23	00:00:28	00:00:05
3.1.202	console	00:00:50	00:00:32	-00:00:18
3.1.302	console	00:00:31	00:00:29	-00:00:02
3.1.407	console	00:00:27	00:00:26	-00:00:01
5.0.104	console	00:00:18	00:00:30	00:00:12
5.0.201	console	00:00:21	00:00:22	00:00:01

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	mstest	00:00:45	00:00:34	-00:00:11
2.1.403	mstest	00:00:31	00:00:31	00:00:00
2.1.522	mstest	00:00:33	00:00:21	-00:00:12
2.1.617	mstest	00:00:25	00:00:26	00:00:01
2.1.701	mstest	00:02:19	00:00:21	-00:01:58
2.1.814	mstest	00:00:42	00:00:31	-00:00:11
3.1.113	mstest	00:00:38	00:00:30	-00:00:08
3.1.202	mstest	00:00:46	00:00:20	-00:00:26
3.1.302	mstest	00:00:51	00:00:42	-00:00:09
3.1.407	mstest	00:00:20	00:00:35	00:00:15
5.0.104	mstest	00:00:45	00:00:35	-00:00:10
5.0.201	mstest	00:00:28	00:00:32	00:00:04

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	xunit	00:00:41	00:00:26	-00:00:15
2.1.403	xunit	00:00:37	00:00:35	-00:00:02
2.1.522	xunit	00:00:34	00:00:21	-00:00:13
2.1.617	xunit	00:00:42	00:00:30	-00:00:12
2.1.701	xunit	00:00:35	00:00:14	-00:00:21
2.1.814	xunit	00:00:48	00:00:19	-00:00:29
3.1.113	xunit	00:00:37	00:00:20	-00:00:17
3.1.202	xunit	00:00:32	00:00:23	-00:00:09
3.1.302	xunit	00:00:50	00:00:18	-00:00:32
3.1.407	xunit	00:00:46	00:00:22	-00:00:24
5.0.104	xunit	00:00:57	00:00:32	-00:00:25
5.0.201	xunit	00:01:11	00:00:35	-00:00:36

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	web	00:00:37	00:00:40	00:00:03
2.1.403	web	00:00:50	00:00:39	-00:00:11
2.1.522	web	00:01:39	00:00:32	-00:01:07
2.1.617	web	00:02:07	00:00:38	-00:01:29
2.1.701	web	00:01:23	00:00:24	-00:00:59
2.1.814	web	00:01:20	00:00:27	-00:00:53
3.1.113	web	00:00:46	00:00:22	-00:00:24
3.1.202	web	00:00:24	00:00:24	00:00:00
3.1.302	web	00:00:31	00:00:25	-00:00:06
3.1.407	web	00:00:30	00:00:25	-00:00:05
5.0.104	web	00:00:30	00:00:22	-00:00:08
5.0.201	web	00:00:14	00:00:20	00:00:06

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	mvc	00:00:34	00:00:37	00:00:03
2.1.403	mvc	00:00:58	00:00:53	-00:00:05
2.1.522	mvc	00:01:04	00:00:37	-00:00:27
2.1.617	mvc	00:01:15	00:00:24	-00:00:51
2.1.701	mvc	00:00:46	00:00:37	-00:00:09
2.1.814	mvc	00:01:09	00:00:29	-00:00:40
3.1.113	mvc	00:00:29	00:00:28	-00:00:01
3.1.202	mvc	00:00:37	00:00:20	-00:00:17
3.1.302	mvc	00:00:39	00:00:27	-00:00:12
3.1.407	mvc	00:00:32	00:00:25	-00:00:07
5.0.104	mvc	00:00:31	00:00:22	-00:00:09
5.0.201	mvc	00:00:27	00:00:24	-00:00:03

.NET SDK Version	AppName	Time with warmup	Time without warmup	Delta
2.1.302	webapi	00:00:29	00:00:45	00:00:16
2.1.403	webapi	00:00:56	00:00:38	-00:00:18
2.1.522	webapi	00:01:30	00:00:23	-00:01:07
2.1.617	webapi	00:00:50	00:00:24	-00:00:26
2.1.701	webapi	00:01:13	00:00:34	-00:00:39
2.1.814	webapi	00:01:05	00:00:25	-00:00:40
3.1.113	webapi	00:00:24	00:00:10	-00:00:14
3.1.202	webapi	00:00:28	00:00:13	-00:00:15
3.1.302	webapi	00:00:27	00:00:20	-00:00:07
3.1.407	webapi	00:00:32	00:00:19	-00:00:13
5.0.104	webapi	00:00:33	00:00:23	-00:00:10
5.0.201	webapi	00:00:22	00:00:20	-00:00:02

@bishal-pdMSFT, Comparing results with warmup and without for Ubuntu Server we find out that there is no much profit performance to run smoke tests during Ubuntu image generation process and this section can be rid of - https://github.com/actions/virtual-environments/blob/main/images/linux/scripts/installers/dotnetcore-sdk.sh.

# Smoke test each SDK
for sdk in $sortedSdks; do
    mksamples "$sdk" "console"
    mksamples "$sdk" "mstest"
    mksamples "$sdk" "xunit"
    mksamples "$sdk" "web"
    mksamples "$sdk" "mvc"
    mksamples "$sdk" "webapi"
done

[bishal-pdMSFT]

Yeah it seems not much benefit for Ubuntu and would be good to skip warmup for Ubuntu.

[bishal-pdMSFT]

For Windows, it seems .NET has made some changes and there is no benefit in 5.x version. You can think of skipping warmup for 5.x on Windows as well.