Scenario-Level Parallelization

[Code-Grump]

Scenario-level parallelization is something that's long been mentioned for SpecFlow and for Reqnroll has come up at least twice recently:

• Parallel execution using Mstest+reqnroll #116
• Parallel scenario tests fail #114

There's clearly a desire to run tests with scenario-level parallelization; I think this might be a solid feature to add, something with real benefits to encourage users to adopt Reqnroll.

Do we have any idea what it would actually take to enable this?

• What do we currently expect to break when trying to run scenarios in parallel?
• Are there any parts of Reqnroll's design that we know will be a problem for parallel invocation?
• For the long-time maintainers: have any attempts been made to address this in the past? Can it teach us anything?

[gasparnagy]

Here is my brain dump regarding this:

• I am not aware of any conceptual problem that would disallow scenario-level parallelism. The fact that it is not working should be considered as misbehavior. (Of course it can turn out that I was wrong.)
• It seemed in the past that the problematic part is BoDi, but earlier when that was a separate project/repo, diagnosing and experimenting with that was hard.
• I don't know if it is relevant, but we (I 😀) have made a design mistake: I assumed that within a scenario execution, there is no need for thread-safety, because that is the most atomic element of test parallelism. Therefore the ScenarioContext and the related BoDi container was not designed to be thread-safe. It turned out however, that the automation code CAN make multiple parallel threads (explicitly or implicitly via async), so there can be cases where this causes an issue. Maybe this is not the most direct problem, but this is going to be a problem as well and might be related. So in short, we have to make sure that the ScenarioContext is thread safe.
• It was also a related problem that the DI resolves the classes from the grouping containers (testrun, testthread, feature) on demand. That means that if a rarely used interface was registered to one of these global containers, they might have been only resolved when the first scenario needed it. If multiple scenarios running in parallel that need this, it might cause strange issues. As far as I know these problems were solved now, but there might have been a few cases when this is still an issue.
• Another problem was that it was hard to make an automated test to verify this. I hope the SystemTests will provide a solution for this now.

As far as I know, @tzongithub did the last attempt to fix parallel issues, maybe he can add further notes.

I would say that probably the strategy should be to pick a test runner (e.g. MsTest) and make a test in the SystemTests that reproduces the error (with a good probability). Once we see that, we can start analyzing and fixing it.

[tulavalle]

Yes, parallelizing at the cear level is very relevant. From what I researched, parallelism is limited at the feature level by the runner, in my case, xunit. Until version 5.0 of Net 5.0 I used specrun, there the parallelism was at the scenario level. Great. However, I needed to change the Net version to 6.0, at the time, as I always need to keep the latest version, but Specrun was discontinued, stopping at Net5.0 I needed to change it and opted for xUnit, however, it and other executors only have parallelism at feature level. A loss. It's been a while since I researched the subject, but I don't think it has changed.

[gasparnagy]

Back in 2019, I wrote a summary about parallel problems. I don't know how much that is relevant but here is the link: SpecFlowOSS/SpecFlow#1535

[Code-Grump]

Relevant: https://microsoft.github.io/coyote/ - a framework for concurrency-testing.

The Coyote testing tool can systematically explore a large number of interleavings of concurrent operations as well as non-deterministic choices so that it covers a large set of behaviors in a very short time.

[gasparnagy]

@obligaron started a PR related to this: #119

[sampath-ganesan]

@obligaron started a PR related to this: #119

HI @gasparnagy, i hope you are doing well. any update on this?

[gasparnagy]

We realized that we need to fix some other aspects first, so it was put on hold. I think now these dependencies are almost fixed so we can pick it up again soon.

[sampath-ganesan]

thanks for the update

[gasparnagy]

Duplicate topic: https://github.com/orgs/reqnroll/discussions/53
I will keep this one as this has more recent conversations.

[Xen0byte]

Hi, I can see that an MSTest implementation is in the works but, if this helps with anything, this is what happens in NUnit. I'm mostly making this comment because MSTest, by default, creates an instance of the test class for each test method in that class, but for NUnit this behaviour needs to be explicitly opted into.

[assembly: Parallelizable(ParallelScope.Children)]

  Message: 
System.NullReferenceException : Object reference not set to an instance of an object.
TearDown : System.NullReferenceException : Object reference not set to an instance of an object.

  Stack Trace: 
StackedInternalContextManager`1.RemoveTop()
ContextManager.CleanupStepContext()
TestExecutionEngine.StepAsync(StepDefinitionKeyword stepDefinitionKeyword, String keyword, String text, String multilineTextArg, Table tableArg)
TestRunner.ThenAsync(String text, String multilineTextArg, Table tableArg, String keyword)
DetokeniserFeature.DetokeniseUntokenisedData() line 56
GenericAdapter`1.GetResult()
AsyncToSyncAdapter.Await(Func`1 invoke)
TestMethodCommand.RunTestMethod(TestExecutionContext context)
TestMethodCommand.Execute(TestExecutionContext context)
<>c__DisplayClass1_0.<Execute>b__0()
BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)
--TearDown
TestExecutionEngine.OnScenarioEndAsync()
TestRunner.OnScenarioEndAsync()
DetokeniserFeature.TestTearDownAsync()
GenericAdapter`1.GetResult()
AsyncToSyncAdapter.Await(Func`1 invoke)
SetUpTearDownItem.RunSetUpOrTearDownMethod(TestExecutionContext context, IMethodInfo method)
SetUpTearDownItem.RunTearDown(TestExecutionContext context)

  Standard Output: 
-> warning: The previous ScenarioContext was not disposed.
-> error: Object reference not set to an instance of an object. (0.0s)
-> error: Object reference not set to an instance of an object. (0.0s)
-> warning: The previous ScenarioContext was already disposed.
-> warning: The previous ScenarioContext was already disposed.

[assembly: FixtureLifeCycle(LifeCycle.InstancePerTestCase)]

  Message: 
OneTimeSetUp: System.InvalidOperationException : Only static OneTimeSetUp and OneTimeTearDown are allowed for InstancePerTestCase mode.

[drosenba]

I'm using the SpecFlow v3 now and have not used parallel. I want to upgrade to Reqnroll and use parallel. I plan to use MSUnit. I need to ensure that parallel is only at the feature level and not at the scenario level. Multiple features can run at the same time, but the scenarios within a feature must run in the order they are within the feature/file. If you make changes to allow scenario-level parallelization would you please ensure that they won't affect being able to to have parallel only at the feature level.
Is there some configuration I can use now (SpecFlow.json or otherwise) that will ensure parallel is only at the feature level even in the future?

[ajeckmans]

AFAIK if you set mstest to parellelize on class level you should be good to go.

[tulavalle]

In MSTest you can inform the level of parallelization, something like: [assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]. In XUnit, for example, by default the parallelism is at the feature level, unfortunately it does not support parallelism at the scenario level. But I emphasize here that in terms of tests one of the premises is that they are totally independent, that is, that it is not necessary to guarantee the order of execution of scenarios, each one of them must be treated independently, after all, if the first test in the order fails, all the others will fail, right? For this, you can adopt the mass data strategy that is most appropriate for the project.

[cburnettADC]

Hello, are there any updates/timeline for this feature to be added? My team's adoption of Reqnroll is highly dependent on scenario level parallelization being possible.

[sudgarre]

Thank you @gasparnagy . I would be happy to test the changes once the PR is merged & also check any impact on Extent Test Reports that we have integrated with reqnroll for scenario parallel execution.

[sampath-ganesan]

as for the extent report this is how it will look like

as you can see it will show two steps same time its because of parallel execution i am not sure what is the root cause for that

[sampath-ganesan]

its getting more weird when i run it in class level parallelization, its generating two different reports for respective features

[maritblom]

@sampath-ganesan I'm seeing the same; scenarios blending together. I'm looking forward to #277 getting merged in @gasparnagy . For reference, I'm using reqnroll 2.1.1, mstest 3.4.3, with .net 8.0.

[sampath-ganesan]

i am using that PR changes still same issue is occurring

[sampath-ganesan]

Hi @gasparnagy, thanks for rising PR. I forked this change and used in my Realtime project (Which is using MSTest provider + UI automation) and found two issues

1. when initializing the web driver we had to use AsyncLocal to initiate the browser this i am doing because when test run in browser its returning same webdriver session id to all scenarios which will do the action in that webdriver session only (Is there anything we can do on reqnroll project to solve this?)
2. when running in parallel run with this change + AsyncLocal i am able to run scenarios in parallel without any issues but the issue starts at the reporting when i see the report i can see the standard out is not having the proper output such as if i run 4 scenarios and all scenarios got executed then only one scenario have the standard output for all 4 scenarios along with thread Id so i did some changes on
   

   Reqnroll/Reqnroll/Tracing/TraceListenerQueue.cs

   Line 54 in cda0318

   if (_isThreadSafeTraceListener || !_testRunnerManager.IsMultiThreaded)

   
   i removed invers condition on this and it started logging the output properly i checked with MSTest only i am not sure how it will behave on other test providers.
   can you please consider this fix as well which will benefit me and my project

[sampath-ganesan]

i can see there webdriver is initializing with Lazy it might be solving the issue

[gasparnagy]

@sampath-ganesan In that case that seems to be a separate issue and not directly related to scenario-level parallelization. Could you please file a separate issue for that?

[sampath-ganesan]

do i need to create different issue for both issue?

[gasparnagy]

No, I think it is enough to have one

[sampath-ganesan]

Issue created #298