Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUREFIRE-2037] Allow for surefire-junit-platform's TestMethodFilter … #487

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
~ KIND, either express or implied. See the License for the
~ specific language governing permissions and limitations
~ under the License.
~ under the License.
-->

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
Expand Down Expand Up @@ -532,8 +532,7 @@
<exclude>src/test/resources/**/*</exclude>
<exclude>src/test/resources/**/*.css</exclude>
<exclude>**/*.jj</exclude>
<exclude>src/main/resources/META-INF/services/org.apache.maven.surefire.api.provider.SurefireProvider
</exclude>
<exclude>src/main/resources/META-INF/services/**</exclude>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't change this POM, it has nothing to do with what you are aiming for. Rather do not make any additional refactoring because it is very hard to separate reasonable and non-reasonable changes. I guess you wanted to make some specific improvement in JUnit5 provider. Do not press CTRL+L because there is again unnecessary changes.

Copy link
Author

@crizzis crizzis Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow.

This was an intentional change, otherwise Rat would expect a license inside META-INF/services/org.apache.maven.surefire.junitplatform.TestSelectorFactory

Should I add an entry here for that specific file instead?

<exclude>DEPENDENCIES</exclude>
<exclude>.m2/**</exclude>
<exclude>.m2</exclude>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -195,13 +195,13 @@ public boolean shouldRun( Class<?> testClass, String methodName )
/**
* Returns {@code true} if satisfies {@code testClassFile} and {@code methodName} filter.
*
* @param testClassFile format must be e.g. "my/package/MyTest.class" including class extension; or null
* @param methodName real test-method name; or null
* @param containerName format must be e.g. "my/package/MyTest.class" including class extension; or null
Copy link
Contributor

@Tibor17 Tibor17 Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't change this class, it has nothing to do with what you are aiming for. Rather do not make any additional refactoring because it is very hard to separate reasonable and non-reasonable changes. I guess you wanted to make some specific improvement in JUnit5 provider. Do not press CTRL+ALT+L because there is again unnecessary changes.

Copy link
Author

@crizzis crizzis Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter names of this method have been changed to reflect their new meaning. The second parameter to this method, for instance, will no longer only ever represent method names. That's the entire purpose of this PR.

Would you care to explain why you would expect the parameters to retain their original names?

* @param selectorName real test-method name; or null
*/
@Override
public boolean shouldRun( String testClassFile, String methodName )
public boolean shouldRun( String containerName, String selectorName )
{
if ( isEmpty() || isBlank( testClassFile ) && isBlank( methodName ) )
if ( isEmpty() || isBlank( containerName ) && isBlank( selectorName ) )
{
return true;
}
Expand All @@ -217,7 +217,7 @@ public boolean shouldRun( String testClassFile, String methodName )
{
for ( ResolvedTest filter : getIncludedPatterns() )
{
if ( filter.matchAsInclusive( testClassFile, methodName ) )
if ( filter.matchAsInclusive( containerName, selectorName ) )
{
shouldRun = true;
break;
Expand All @@ -229,7 +229,7 @@ public boolean shouldRun( String testClassFile, String methodName )
{
for ( ResolvedTest filter : getExcludedPatterns() )
{
if ( filter.matchAsExclusive( testClassFile, methodName ) )
if ( filter.matchAsExclusive( containerName, selectorName ) )
{
shouldRun = false;
break;
Expand Down
41 changes: 41 additions & 0 deletions surefire-providers/surefire-junit-platform/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,47 @@
</includes>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>api-jar</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<classifier>api</classifier>
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wasn't sure whether to classify this JAR with api or spi.

In any case, this is simply a slim version of the JAR for service providers who otherwise do not depend on surefire-junit-platform

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a hack. Why you have introduced Apache SPI and you change the default behavior this way? JUnit5 SPI do not exist with same purpose? The plugin has configuration of class/methods. If somebody changes this in the SPI, then what you be valid? Configuration or SPI? Pls describe very properly what you are aiming for in the JIRA in details.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are other two SPI modules surefire-extensions-spi (for the forked JVM) and surefire-extensions-api (for the plugin process) but nothing is suited for this scenario of providers.
Can you explain how the customer would implement the interface TestSelectorFactory you have introduced? What types of sources can be filtered and how they can be configured?

Copy link
Author

@crizzis crizzis Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain how the customer would implement the interface TestSelectorFactory you have introduced? What types of sources can be filtered and how they can be configured?

Absolutely.

Here is my implementation for ArchUnit's FieldSource:

package com.tngtech.archunit.junit.surefire;

import com.tngtech.archunit.junit.FieldSource;
import org.apache.maven.surefire.junitplatform.TestSelectorFactory;
import org.junit.platform.engine.TestSource;

public class FieldSelectorFactory implements TestSelectorFactory {
    @Override
    public boolean supports(TestSource testSource) {
        return testSource instanceof FieldSource;
    }

    @Override
    public String getContainerName(TestSource testSource) {
        return ((FieldSource) testSource).getClassName();
    }

    @Override
    public String getSelectorName(TestSource testSource) {
        return ((FieldSource) testSource).getFieldName();
    }
}

It solves the problem of adding support for applying the same filtering logic Surefire now applies to methods, to other test sources (FieldSource in this case).

In case you're unfamiliar with ArchUnit, here's what an ArchUnit test might look like:

@AnalyzeClasses(packages = "com.tngtech.archunit.example.layers")
public class CodingRulesTest {
    
    @ArchTest
    private final ArchRule loggers_should_be_private_static_final =
            fields().that().haveRawType(Logger.class)
                    .should().bePrivate()
                    .andShould().beStatic()
                    .andShould().beFinal()
                    .because("we agreed on this convention");
}

As you can see, test cases are represented by fields, and not methods. This is also why the default Surefire include / exclude filters do not work.

JUnit5 SPI do not exist with same purpose?

Yes, of course it exists, and that's exactly what ArchUnit does under the hood. There's no way to influence Surefire filtering logic from inside a JUnit extension, though.

I've already verified providing this implementation of the new SPI in ArchUnit solves the linked ArchUnit issue for Surefire. I'm now about to suggest similar extensibility to other build tools.

Copy link
Author

@crizzis crizzis Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, if I understand correctly, here:

The plugin has configuration of class/methods. If somebody changes this in the SPI, then what you be valid? Configuration or SPI?

You're asking what happens if someone overwrites the service to something else than MethodSelectorFactory. The answer is here:

private Collection<TestSelectorFactory> loadSelectorFactories()
    {
        return StreamSupport.stream( ServiceLoader.load( TestSelectorFactory.class ).spliterator(), false )
            .collect( Collectors.toSet() );
    }

Basically, there's no 'changing the SPI'. MethodSelectorFactory is always provided as the default implementation, and extending libraries can only add new service implementations. MethodSelectorFactory cannot be resigned from, though.

Thus, for those users who do not extend this SPI in any way, the behavior of Surefire will be consistent with the behavior before this PR.

There are other two SPI modules surefire-extensions-spi (for the forked JVM) and surefire-extensions-api (for the plugin process)

Yes, and unfortunately, they do not fit my use case. I could implement a custom SurefireProvider, but that would basically mean re-implementing the current JUnitPlatformProvider, which is 99% suitable but completely closed for extension. This minuscule SPI introduction is really the smallest possible change that solves the problem without affecting other plugin users.

<includes>
<include>org/apache/maven/surefire/junitplatform/TestSelectorFactory.class</include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>build-helper-maven-plugin</artifactId>
<version>3.2.0</version>
<executions>
<execution>
<id>api-jar</id>
<goals>
<goal>attach-artifact</goal>
</goals>
<configuration>
<artifacts>
<artifact>
<classifier>api</classifier>
<type>jar</type>
<file>${project.build.directory}/${project.artifactId}-${project.version}-api.jar</file>
</artifact>
</artifacts>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,17 @@
import java.io.StringReader;
import java.io.UncheckedIOException;
import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.Properties;
import java.util.ServiceLoader;
import java.util.logging.Logger;
import java.util.stream.Collectors;
import java.util.stream.StreamSupport;

import org.apache.maven.surefire.api.provider.AbstractProvider;
import org.apache.maven.surefire.api.provider.ProviderParameters;
Expand Down Expand Up @@ -105,7 +109,7 @@ public JUnitPlatformProvider( ProviderParameters parameters )
public Iterable<Class<?>> getSuites()
{
try
{
{
return scanClasspath();
}
finally
Expand Down Expand Up @@ -235,7 +239,7 @@ private void execute( TestsToRun testsToRun, RunListenerAdapter adapter )
} );
}
}

private void closeLauncher()
{
if ( launcher instanceof AutoCloseable )
Expand Down Expand Up @@ -263,6 +267,12 @@ private LauncherDiscoveryRequest buildLauncherDiscoveryRequestForRerunFailures(
return builder.build();
}

private Collection<TestSelectorFactory> loadSelectorFactories()
{
return StreamSupport.stream( ServiceLoader.load( TestSelectorFactory.class ).spliterator(), false )
.collect( Collectors.toSet() );
}

private Filter<?>[] newFilters()
{
List<Filter<?>> filters = new ArrayList<>();
Expand All @@ -278,7 +288,7 @@ private Filter<?>[] newFilters()
TestListResolver testListResolver = parameters.getTestRequest().getTestListResolver();
if ( !testListResolver.isEmpty() )
{
filters.add( new TestMethodFilter( testListResolver ) );
filters.add( new TestSelectorFilter( testListResolver, loadSelectorFactories() ) );
}

getPropertiesList( INCLUDE_JUNIT5_ENGINES_PROP )
Expand All @@ -289,7 +299,7 @@ private Filter<?>[] newFilters()
.map( EngineFilter::excludeEngines )
.ifPresent( filters::add );

return filters.toArray( new Filter<?>[ filters.size() ] );
return filters.toArray( new Filter<?>[ 0 ] );
}

Filter<?>[] getFilters()
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
package org.apache.maven.surefire.junitplatform;

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import org.junit.platform.engine.TestSource;
import org.junit.platform.engine.support.descriptor.MethodSource;

/**
* The default implementation for {@link TestSelectorFactory}, recognizing instances of {@link MethodSource}
*/
public class MethodSelectorFactory implements TestSelectorFactory
{
@Override
public boolean supports( TestSource source )
{
return source instanceof MethodSource;
}

@Override
public String getContainerName( TestSource source )
{
return ( ( MethodSource ) source ).getClassName();
}

@Override
public String getSelectorName( TestSource source )
{
return ( ( MethodSource ) source ).getMethodName();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
package org.apache.maven.surefire.junitplatform;

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

import org.apache.maven.surefire.api.testset.TestListResolver;
import org.junit.platform.engine.TestSource;

import java.util.Objects;

/**
* A test selector factory used in combination with a {@link org.apache.maven.surefire.api.testset.TestListResolver}
* to determine whether a given {@link org.junit.platform.engine.TestSource} should be considered for running.
*
* <br><br><p>This is a service provider interface; clients may provide their own implementations
* that will be applied along the default {@link MethodSelectorFactory}</p>
*/
public interface TestSelectorFactory
{

boolean supports( TestSource source );

String getContainerName( TestSource source );

String getSelectorName( TestSource source );

default boolean isClassContainer()
{
return true;
}

default TestSelector createSelector( TestSource source )
{
String containerName = getContainerName( source );
return new TestSelector(
isClassContainer() ? TestListResolver.toClassFileName( containerName ) : containerName,
getSelectorName( source ) );
}

/**
* Represents a single test case selector
* (e.g. a fully-qualified class name + test-annotated method name)
*/
class TestSelector
{
private final String containerName;
private final String selectorName;

public TestSelector( String containerName, String selectorName )
{
this.containerName = containerName;
this.selectorName = selectorName;
}

public String getContainerName()
{
return containerName;
}

public String getSelectorName()
{
return selectorName;
}

@Override
public boolean equals( Object o )
{
if ( this == o )
{
return true;
}
if ( o == null || getClass() != o.getClass() )
{
return false;
}
TestSelector that = ( TestSelector ) o;
return containerName.equals( that.containerName ) && Objects.equals( selectorName, that.selectorName );
}

@Override
public int hashCode()
{
return Objects.hash( containerName, selectorName );
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -22,39 +22,48 @@
import org.apache.maven.surefire.api.testset.TestListResolver;
import org.junit.platform.engine.FilterResult;
import org.junit.platform.engine.TestDescriptor;
import org.junit.platform.engine.support.descriptor.MethodSource;
import org.junit.platform.engine.TestSource;
import org.junit.platform.launcher.PostDiscoveryFilter;

import java.util.Collection;
import java.util.Optional;

/**
* @since 2.22.0
*/
class TestMethodFilter
class TestSelectorFilter
implements PostDiscoveryFilter
{

private final TestListResolver testListResolver;
private final Collection<TestSelectorFactory> selectorFactories;

TestMethodFilter( TestListResolver testListResolver )
TestSelectorFilter( TestListResolver testListResolver, Collection<TestSelectorFactory> selectorFactories )
{
this.testListResolver = testListResolver;
this.selectorFactories = selectorFactories;
}

@Override
public FilterResult apply( TestDescriptor descriptor )
{
boolean shouldRun = descriptor.getSource()
.filter( MethodSource.class::isInstance )
.map( MethodSource.class::cast )
.map( this::shouldRun )
.orElse( true );

.flatMap( source -> resolveFactory( source )
.map( factory -> factory.createSelector( source ) ) )
.map( this::shouldRun )
.orElse( true );
return FilterResult.includedIf( shouldRun );
}

private boolean shouldRun( MethodSource source )
private boolean shouldRun( TestSelectorFactory.TestSelector selector )
{
return this.testListResolver.shouldRun( selector.getContainerName(), selector.getSelectorName() );
}

private Optional<TestSelectorFactory> resolveFactory( TestSource source )
{
String testClass = TestListResolver.toClassFileName( source.getClassName() );
String testMethod = source.getMethodName();
return this.testListResolver.shouldRun( testClass, testMethod );
return selectorFactories.stream()
.filter( factory -> factory.supports( source ) )
.findAny();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
org.apache.maven.surefire.junitplatform.MethodSelectorFactory
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ public static Test suite()
TestSuite suite = new TestSuite();
suite.addTest( new JUnit4TestAdapter( JUnitPlatformProviderTest.class ) );
suite.addTest( new JUnit4TestAdapter( RunListenerAdapterTest.class ) );
suite.addTest( new JUnit4TestAdapter( TestMethodFilterTest.class ) );
suite.addTest( new JUnit4TestAdapter( TestSelectorFilterTest.class ) );
suite.addTest( new JUnit4TestAdapter( TestPlanScannerFilterTest.class ) );
return suite;
}
Expand Down
Loading