Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SUREFIRE-2152] Include name of JUnit5 templated-tests in test description #615

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

s-rwe
Copy link

@s-rwe s-rwe commented Feb 21, 2023

Suggestion to support SUREFIRE-2152, helping developers identify problematic invocations of templated tests much quicker, especially also based on the console output.

The initial commit a87457b to handle JUnit5 templated tests quite separately in RunListenerAdapter also fixes another issue mentioned recently on SUREFIRE-2087, namely this one:

I could assume that especially ff69448 might be controversial in this, since it mixes up the separation between the methodDesc and methodDisp (introduced in SUREFIRE-1546, AFAICS) to some extent. However, I would argue that it's quite important to be able to see the display-name of templated tests in general (i.e., in particular also in the default output of the console reporter), else it can be tedious to figure out more details.

Still, it might be useful to make the changes of ff69448 conditional of a configuration flag - i.e. keeping it disabled by default and have a flag for opt-in (I don't really know how a config-flag would need to be done, help welcome if you'd consider that useful too)

Validation:

Failed to install project dependencies: MavenProject: org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M10-SNAPSHOT
[...]
cannot install /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom to same path

…displayName

Ensures that all JUnit5 templated tests have a unique name that includes
their invocation ID (index)

This change also covers an issue of SUREFIRE-2087 where certain tests
(namely, tests defined through a @testtemplate) still cause mixups
of test invocations in relation with rerunFailingTestsCount, which
can be mixed up by surefire if their name isn't unique for each distinct
invocation.

For an example description of this issue, see also:
https://issues.apache.org/jira/browse/SUREFIRE-2087#comment-17690951
…iter

It seems cleaner to strip the suffix of templated tests again here,
since it is unrelated to the actual method signature, and might hence
just cause confusion in context of a StackTraceWriter.
…hodDesc

This includes the displayName (e.g. value of name="..." of a
@ParameterizedTest) in the method description / test identifier.

Provides more information than merely the invocation ID in case
of failing tests, also for the console reporter. This can
(especially if the actual list of arguments are quite large and/or
dynamically determined at runtime) help a lot to identify which
actual cases are the problematic ones.
… changes

Since the name of templated tests now includes the displayName as suffix,
the nameText is the same (and hence set to null) in some test-cases
@slawekjaranowski
Copy link
Member

Please check result of ITs - and fix ITs or code according

@s-rwe
Copy link
Author

s-rwe commented Feb 26, 2023

Please check result of ITs - and fix ITs or code according

I'd be happy to, but I'm unable to run the ITs locally to do this in the first place (as mentioned above in my initial description)

Couple of things I did:

  • Followed the steps outlined https://github.com/apache/maven-surefire#development-information
  • Disabled rat-checks with -Drat.skip=true (those also failed quickly at first)
  • mvn -Prun-its clean install always fails, as mentioned before in the PR description
  • Same failure also happens for me on master (namely, I tried it on 208eae2) - so unrelated to the changes of this PR
  • Tried a few JDKs too (namely, corretto 19 and 1.8)

Not to speak of trying to get specific tests to run in IntelliJ to get them fixed (I have a different set of issues here) - but would be nice to at least be able to run things from commandline via mvn.

So I'm kind of stuck on that end, tips / hints anyone?

@slawekjaranowski
Copy link
Member

Can you show me more logs for your erro:

Failed to install project dependencies: MavenProject: org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M10-SNAPSHOT
[...]
cannot install /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom to same path

Which plugin was executed for it?

@s-rwe
Copy link
Author

s-rwe commented Mar 9, 2023

Can you show me more logs for your erro:

Failed to install project dependencies: MavenProject: org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M10-SNAPSHOT
[...]
cannot install /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom to same path

Which plugin was executed for it?

Sure, here is some more log-output if that helps. Note that I once more ran this on a master / 208eae2 (git-cleaned, and with .m2/repository cleaned out as well) - so not directly related to the branch of this PR.

$ mvn -Drat.skip=true -Prun-its clean install

[...]

[INFO]
[INFO] --- maven-jar-plugin:3.2.2:jar (default-jar) @ maven-failsafe-plugin ---
[INFO] Building jar: /development/projects/3rdparty/maven-surefire/maven-failsafe-plugin/target/maven-failsafe-plugin-3.0.0-M10-SNAPSHOT.jar
[INFO]
[INFO] --- maven-plugin-plugin:3.6.4:addPluginArtifactMetadata (default-addPluginArtifactMetadata) @ maven-failsafe-plugin ---
[INFO]
[INFO] --- maven-site-plugin:3.11.0:attach-descriptor (attach-descriptor) @ maven-failsafe-plugin ---
[INFO] Skipping because packaging 'maven-plugin' is not pom.
[INFO]
[INFO] --- maven-invoker-plugin:3.2.2:install (pre-its) @ maven-failsafe-plugin ---
[INFO] Installing /development/projects/3rdparty/maven-surefire/maven-surefire-common/pom.xml to /home/me/.m2/repository/org/apache/maven/surefire/maven-surefire-common/3.0.0-M10-SNAPSHOT/maven-surefire-common-3.0.0-M10-SNAPSHOT.pom
[INFO] Installing /development/projects/3rdparty/maven-surefire/maven-surefire-common/target/maven-surefire-common-3.0.0-M10-SNAPSHOT.jar to /home/me/.m2/repository/org/apache/maven/surefire/maven-surefire-common/3.0.0-M10-SNAPSHOT/maven-surefire-common-3.0.0-M10-SNAPSHOT.jar
[INFO] Installing /development/projects/3rdparty/maven-surefire/maven-surefire-common/pom.xml to /home/me/.m2/repository/org/apache/maven/surefire/maven-surefire-common/3.0.0-M10-SNAPSHOT/maven-surefire-common-3.0.0-M10-SNAPSHOT.pom
[INFO] Installing /development/projects/3rdparty/maven-surefire/pom.xml to /home/me/.m2/repository/org/apache/maven/surefire/surefire/3.0.0-M10-SNAPSHOT/surefire-3.0.0-M10-SNAPSHOT.pom
[INFO] Installing /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom to /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Maven Surefire 3.0.0-M10-SNAPSHOT:
[INFO]
[INFO] Apache Maven Surefire .............................. SUCCESS [  1.616 s]
[INFO] Surefire Shared Utils .............................. SUCCESS [  0.993 s]
[INFO] SureFire Logger API ................................ SUCCESS [  1.685 s]
[INFO] SureFire API ....................................... SUCCESS [ 14.029 s]
[INFO] Surefire Extensions API ............................ SUCCESS [  1.651 s]
[INFO] Surefire Extensions SPI ............................ SUCCESS [  0.520 s]
[INFO] SureFire Booter .................................... SUCCESS [  7.134 s]
[INFO] Maven Surefire Test-Grouping Support ............... SUCCESS [  1.370 s]
[INFO] SureFire Providers ................................. SUCCESS [  0.222 s]
[INFO] Shared JUnit3 Provider Code ........................ SUCCESS [  1.096 s]
[INFO] Shared Java 5 Provider Base ........................ SUCCESS [  1.464 s]
[INFO] Shared JUnit4 Provider Code ........................ SUCCESS [  1.431 s]
[INFO] Shared JUnit48 Provider Code ....................... SUCCESS [  2.059 s]
[INFO] SureFire JUnit Runner .............................. SUCCESS [  1.320 s]
[INFO] SureFire JUnit4 Runner ............................. SUCCESS [  1.289 s]
[INFO] Maven Surefire Common .............................. SUCCESS [ 40.743 s]
[INFO] SureFire JUnitCore Runner .......................... SUCCESS [01:59 min]
[INFO] SureFire JUnit Platform Runner ..................... SUCCESS [  2.780 s]
[INFO] SureFire TestNG Utils .............................. SUCCESS [  1.215 s]
[INFO] SureFire TestNG Runner ............................. SUCCESS [  2.029 s]
[INFO] ShadeFire JUnit3 Provider .......................... SUCCESS [  0.576 s]
[INFO] Surefire Report Parser ............................. SUCCESS [  1.514 s]
[INFO] Maven Surefire Plugin .............................. SUCCESS [  2.605 s]
[INFO] Maven Failsafe Plugin .............................. FAILURE [  2.804 s]
[INFO] Maven Surefire Report Plugin ....................... SKIPPED
[INFO] Maven Surefire Integration Tests ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:31 min
[INFO] Finished at: 2023-03-09T09:18:15+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-invoker-plugin:3.2.2:install (pre-its) on project maven-failsafe-plugin: Failed to install project dependencies: MavenProject: org.apache.maven.plugins:maven-failsafe-plugin:3.0.0-M10-SNAPSHOT @ /development/projects/3rdparty/maven-surefire/maven-failsafe-plugin/pom.xml: Failed to install project parents: MavenProject: org.apache.maven.surefire:maven-surefire-common:3.0.0-M10-SNAPSHOT @ /development/projects/3rdparty/maven-surefire/maven-surefire-common/pom.xml: Failed to install POM: MavenProject: org.apache.maven:maven-parent:36 @ /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom: Failed to install artifact: org.apache.maven:maven-parent:pom:36: Failed to install artifact org.apache.maven:maven-parent:pom:36: cannot install /home/me/.m2/repository/org/apache/maven/maven-parent/36/maven-parent-36.pom to same path -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :maven-failsafe-plugin

Version / Java - info as well if relevant:

$ mvn --version
Apache Maven 3.8.2 (ea98e05a04480131370aa0c110b8c54cf726c06f)
Maven home: /software/apache-maven-3.8.2
Java version: 19.0.2, vendor: Amazon.com Inc., runtime: /usr/lib/jvm/java-19-amazon-corretto
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.13.0-52-generic", arch: "amd64", family: "unix"

@slawekjaranowski
Copy link
Member

very strange ... when I build from master I have different plugin versions:

INFO] --- jar:3.3.0:jar (default-jar) @ maven-failsafe-plugin ---
[INFO] Building jar: /Users/slawomir.jaranowski/repos/apache/maven-surefire/maven-failsafe-plugin/target/maven-failsafe-plugin-3.0.0-M10-SNAPSHOT.jar
[INFO] 
[INFO] --- plugin:3.7.0:addPluginArtifactMetadata (default-addPluginArtifactMetadata) @ maven-failsafe-plugin ---
[INFO] 
[INFO] --- site:3.12.1:attach-descriptor (attach-descriptor) @ maven-failsafe-plugin ---
[INFO] Skipping because packaging 'maven-plugin' is not pom.
[INFO] 
[INFO] --- invoker:3.3.0:install (pre-its) @ maven-failsafe-plugin ---

maven-invoker-plugin in my build is 3.3.0 in your 3.2.2

so I assume that you have not actual code in your workspace

Please rebase and resolve conflicts.

@s-rwe
Copy link
Author

s-rwe commented Mar 9, 2023

very strange ... when I build from master I have different plugin versions:

INFO] --- jar:3.3.0:jar (default-jar) @ maven-failsafe-plugin ---
[INFO] Building jar: /Users/slawomir.jaranowski/repos/apache/maven-surefire/maven-failsafe-plugin/target/maven-failsafe-plugin-3.0.0-M10-SNAPSHOT.jar
[INFO] 
[INFO] --- plugin:3.7.0:addPluginArtifactMetadata (default-addPluginArtifactMetadata) @ maven-failsafe-plugin ---
[INFO] 
[INFO] --- site:3.12.1:attach-descriptor (attach-descriptor) @ maven-failsafe-plugin ---
[INFO] Skipping because packaging 'maven-plugin' is not pom.
[INFO] 
[INFO] --- invoker:3.3.0:install (pre-its) @ maven-failsafe-plugin ---

maven-invoker-plugin in my build is 3.3.0 in your 3.2.2

so I assume that you have not actual code in your workspace

Please rebase and resolve conflicts.

I'd rebase the branch of this PR once I'm able to run the ITs in some form (as said that's not working on master either for me at all, would need to sort this out first).

And for running the ITs, I tried the same command again on latest master state (2709f76), same result / same failure.

I also tried running it again with maven-invoker-plugin explicitly set to 3.3.0 in case that would help (i.e. manually by changing maven-failsafe-plugin/pom.xml) -> while I do see that maven picks the changed plugin version up with this, yet again, same result / same failure... :/

Can this be related to the maven version?

@slawekjaranowski
Copy link
Member

OK I have the same error in Maven 3.8.2 - so please upgrade Maven first

@michael-o
Copy link
Member

@s-rwe Do you want to pick this up again?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants