The [Collection] attribute is ignored when running unit tests from contextual menu in the "code view".

[GillesTourreau]

To reproduce the bug, I have created the following unit tests class which contains 4 unit tests that must not be executed in parallel, for that I added the [Collection("Foobar")] attribute on the unit test class.
I added in this unit test class some assertions to check that unit tests are not executed in parallel.

namespace XunitParallelExecutionVsRunnerBug
{
    [Collection("Foobar")]
    public class UnitTest1
    {
        private static readonly bool[] TestRun = new bool[typeof(UnitTest1).GetMethods().Length];

        [Fact]
        public async Task Test1()
        {
            StartTestRun(1);

            await Task.Delay(3000);

            EndTestRun(1);
        }

        [Fact]
        public async Task Test2()
        {
            StartTestRun(2);

            await Task.Delay(3000);

            EndTestRun(2);
        }
        
        [Fact]
        public async Task Test3()
        {
            StartTestRun(3);

            await Task.Delay(3000);

            EndTestRun(3);
        }

        [Fact]
        public async Task Test4()
        {
            StartTestRun(4);

            await Task.Delay(3000);

            EndTestRun(4);
        }

        private static void StartTestRun(int number)
        {
            lock (TestRun)
            {
                // Assert no other tests is running.
                for (var i = 0; i < TestRun.Length; i++)
                {
                    Assert.False(TestRun[i], $"The Test{i + 1} is running in parallel with the Test{number}...");
                }

                TestRun[number - 1] = true;
            }
        }

        private static void EndTestRun(int number)
        {
            lock (TestRun)
            {
                TestRun[number - 1] = false;
            }
        }
    }
}

When I run the unit tests from the Test Explorer, we can see that all the unit tests are executed sequentially.

But when I run the unit tests from the code with the contextual menu (when right click on empty code line between two unit test methods), the unit tests are executed in parallel:

[bradwilson]

Can you provide a repro project? I'm not able to repro this (it passes in both scenarios).

Please include the version of Visual Studio you're using and the results of running dotnet --info.

Thanks!

[GillesTourreau]

@bradwilson, I created a repo with a project to reproduce the bug: https://github.com/GillesTourreau/visualstudio.xunit-ParallelExecutionVsRunnerBug

An other scenario to reproduce the error:

• Clean the output of the project (right click on the project and select Clean).
• Run the unit test (right click between 2 test methods and select Run Tests) => Normally the unit test are executed in sequential.
• Remove the Test4 method (just cut the code in the clipboard).
• Run the unit test (right click between 2 test methods and select Run Tests) => Normally the unit test are executed in sequential.
• Restore the code of the Test4 from the clipboard.
• Run the unit test (right click between 2 test methods and select Run Tests) => Now the unit tests execution are failed, because they are executed in parallel.

Also, you can increase the value of the Task.Delay() to 3 or 5 seconds to see the parallel or sequential execution (I use 1 second, because to reduce the time of the gif video recording).

About my CPU, I have a 12th Gen Intel(R) Core(TM) i9-12900H 2.50 GHz (14 Cores, 20 logical processors recognized by Windows).

I use the last VS version (17.11.5).
The result of dotnet --info command:

.NET SDK:
 Version:           8.0.403
 Commit:            c64aa40a71
 Workload version:  8.0.400-manifests.e99c892e
 MSBuild version:   17.11.9+a69bbaaf5

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22631
 OS Platform: Windows
 RID:         win-x64
 Base Path:   C:\Program Files\dotnet\sdk\8.0.403\

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
 [aspire]
   Installation Source: VS 17.11.35327.3
   Manifest Version:    8.1.0/8.0.100
   Manifest Path:       C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.1.0\WorkloadManifest.json
   Install Type:        FileBased


Host:
  Version:      8.0.10
  Architecture: x64
  Commit:       81cabf2857

.NET SDKs installed:
  8.0.403 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 5.0.17 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.35 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 8.0.10 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download