Output diff of expected and actual values in `ConsoleLauncher` #4017

XJ114514 · 2024-09-23T12:09:56Z

Overview

Current PR can successfully build and generate `junit-platform-console-standalone-1.12.0-SNAPSHOT.jar` containing diff outputs for two different Strings to the console.

Changes made in this draft PR:

add diff function to junit-platform-console/.../tasks/ConsoleTestExecutor.java (output of console)
add dependency to libs.versions.toml and junit-platform-console.gradle.kts of junit-platform-console
add changed that will be made to release-notes-5.12.0-M1.adoc

Tasks that need to finish

add package documentation to junit-platform-console/.../module-info.java (causing testing failure because the auto test cases can not find the diff package)
add auto test cases for diff output in ConsoleTestExecutorTests.java

I hereby agree to the terms of the JUnit Contributor License Agreement.

Definition of Done

There are no TODOs left in the code
Method preconditions are checked and documented in the method's Javadoc
Coding conventions (e.g. for logging) have been followed
Change is covered by automated tests including corner cases, errors, and exception handling
Public API has Javadoc and @API annotations
Change is documented in the User Guide and Release Notes

continue from #3397
Issue: #3139

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence To do list: 1. Add diff function dependency 2. Using diff to generate the output desired Issue: junit-team#3139

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired Issue: junit-team#3139

delete one line of extra code

Fixed different profile issue. Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired ToDo: 1.Automatic test case for diff output Issue: junit-team#3139

Issue: junit-team#3139

Add external diff module to module documentation For unknown reason, the test cases can not find the external module while the gradle can build normally. Issue: junit-team#3139

marcphilipp · 2024-09-23T13:30:09Z

@XJ114514 Could you please change this PR to continue where #3397 left off? We can then tackle the problems that remain one by one.

XJ114514 · 2024-09-24T11:55:33Z

@XJ114514 Could you please change this PR to continue where #3397 left off? We can then tackle the problems that remain one by one.

I apologize for the trouble. But you already superseded and closed #3397. Is there anything I need to do?

marcphilipp · 2024-09-24T12:54:38Z

I closed it because the original author didn't respond anymore. But if you're interested you could create a new branch in your repo based on their work and make the changes that were requested in #3397. Please let me know whether you want to proceed that way.

XJ114514 · 2024-09-24T19:05:12Z

I prefer staying in this PR. The primary goal is to output diff info in the console, create test cases, and successfully merge them.

The extra changes requested in #3397:

Shadowing the original external function
Using theme to highlight the diff info
spreading diff info to all the Junit API output

should come later as it would be easier once we have one successful example.

marcphilipp

The extra changes requested in #3397:

Shadowing the original external function
Using theme to highlight the diff info
spreading diff info to all the Junit API output

should come later as it would be easier once we have one successful example.

We usually only merge complete PRs that leave main ready to be released. But I can help with the shadowing.

gradle/libs.versions.toml

marcphilipp · 2024-09-25T06:15:16Z

junit-platform-console/src/main/java/org/junit/platform/console/tasks/ConsoleTestExecutor.java

@@ -180,11 +186,39 @@ private Optional<TestExecutionListener> createXmlWritingListener(PrintWriter out
 	private void printSummary(TestExecutionSummary summary, PrintWriter out) {
 		// Otherwise the failures have already been printed in detail
 		if (EnumSet.of(Details.NONE, Details.SUMMARY, Details.TREE).contains(outputOptions.getDetails())) {
+			//adding diff code here


This should be done in place where we print the exceptions (as it is in #3397). Otherwise, it would be difficult to find the test that belongs to the diff in case of multiple failing tests.

Which part do you wish to change? The first print happens in DefaultLaucher and the second happens in MutableTestExecutionSummary. Both are in junit-platform-laucher module. Changes will affect all Junit API output I believe?
I can try to work on it over the weekend if you wish.
At the same time, can you help to find why the package documentation test case fails (Test cases can not find diff package once I add it to the module-info.java for documentation)

We also need to make sure that the diff is printed if a details mode other than Details.NONE, Details.SUMMARY, or Details.TREE is used. We should be able to extract the diff-printing code to a separate class and call it from MutableTestExecutionSummary, and the other listeners (FlatPrintingListener etc.).

We also need to make sure that the diff is printed if a details mode other than Details.NONE, Details.SUMMARY, or Details.TREE is used. We should be able to extract the diff-printing code to a separate class and call it from MutableTestExecutionSummary, and the other listeners (FlatPrintingListener etc.).

The solutions I can think of:

add diff output function to junit-platform-commons...util.StringUtils

Use the diff output function in FlatPrintingListener, TreePrintingListener, VerboseTreePrintingListener and MutableTestExecutionSummary
I will start to work on it if the solution looks right for you

add diff output function to junit-platform-commons...util.StringUtils

I think it should stay in junit-platform-console. Please create a separate DiffPrinter class or similar.

DiffPrinter class has been added.

Marked the location of adding diff to FlatPrintingListener and VerboseTreePrintingListener

Will the diff info be generated after the stack trace of the exception?

1. Add diffPrinter class 2. replace diff in consoleTestExecutor with diffPrinter - will be deleted evetually 3. mark the location of adding diff in flatPrinter and verboseTreePrinter. Issue: junit-team#3139

marcphilipp

I think the diffs need to reference the failures, i.e. something like this (where (2) and (3) reference failures from the list above):

Failures (3):
  (1) JUnit Jupiter:AssertionsDemo:groupedAssertions()
    MethodSource [className = 'example.AssertionsDemo', methodName = 'groupedAssertions', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
       example.AssertionsDemo.groupedAssertions(AssertionsDemo.java:52)
  (2) JUnit Jupiter:AssertionsDemo:dependentAssertions()
    MethodSource [className = 'example.AssertionsDemo', methodName = 'dependentAssertions', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: expected: <a
b
c> but was: <b
c
d>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
       example.AssertionsDemo.dependentAssertions(AssertionsDemo.java:63)
  (3) JUnit Jupiter:AssertionsDemo:standardAssertions()
    MethodSource [className = 'example.AssertionsDemo', methodName = 'standardAssertions', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: expected: <fooooooo> but was: <booooooy>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
       example.AssertionsDemo.standardAssertions(AssertionsDemo.java:41)

Diffs (Markdown):
  (2) JUnit Jupiter:AssertionsDemo:groupedAssertions()
   | expected | actual |
   | -------- | ------ |
   | ~~a~~ | b |
   | b | c |
   | c | **d** |
  (3) JUnit Jupiter:AssertionsDemo:standardAssertions()
   | expected | actual |
   | -------- | ------ |
   | ~~f~~oooooo~~o~~ | **b**oooooo**y** |

WDYT?

marcphilipp · 2024-10-08T17:08:19Z

junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java

+	static void printDiff(PrintWriter out, String expected, String actual) {
+		DiffRowGenerator generator = DiffRowGenerator.create().showInlineDiffs(true).inlineDiffByWord(true).oldTag(
+			f -> "~").newTag(f -> "**").build();
+		List<DiffRow> rows = generator.generateDiffRows(Arrays.asList(expected), Arrays.asList(actual));


Don't we have to split the expected and actual Strings into lines here?

What the difference is if we split them into lines?

I wasn't sure. I haven't used this library before and I was just curious why it expects lists of Strings. Do you know the reason?

I don't know the details but all their methods use List<String> as input.

junit-platform-console/src/main/java/org/junit/platform/console/tasks/DiffPrinter.java

…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>

Only if one word has space, show inlineDiffByWord.

XJ114514 · 2024-10-12T17:45:04Z

I think the diffs need to reference the failures, i.e. something like this (where (2) and (3) reference failures from the list above):

Failures (3):
  (1) JUnit Jupiter:AssertionsDemo:groupedAssertions()
    MethodSource [className = 'example.AssertionsDemo', methodName = 'groupedAssertions', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: expected: <2> but was: <3>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:145)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:531)
       example.AssertionsDemo.groupedAssertions(AssertionsDemo.java:52)
  (2) JUnit Jupiter:AssertionsDemo:dependentAssertions()
    MethodSource [className = 'example.AssertionsDemo', methodName = 'dependentAssertions', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: expected: <a
b
c> but was: <b
c
d>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
       example.AssertionsDemo.dependentAssertions(AssertionsDemo.java:63)
  (3) JUnit Jupiter:AssertionsDemo:standardAssertions()
    MethodSource [className = 'example.AssertionsDemo', methodName = 'standardAssertions', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: expected: <fooooooo> but was: <booooooy>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1145)
       example.AssertionsDemo.standardAssertions(AssertionsDemo.java:41)

Diffs (Markdown):
  (2) JUnit Jupiter:AssertionsDemo:groupedAssertions()
   | expected | actual |
   | -------- | ------ |
   | ~~a~~ | b |
   | b | c |
   | c | **d** |
  (3) JUnit Jupiter:AssertionsDemo:standardAssertions()
   | expected | actual |
   | -------- | ------ |
   | ~~f~~oooooo~~o~~ | **b**oooooo**y** |

WDYT?

I can only get the test's display name (the actual test method name). In MutuableTestExecutionSummary, the rest info is from the parent(s) of the testIdentifier. I don't see a simple way to do so in other classes. WDYT?

marcphilipp · 2024-10-14T13:45:27Z

If we list them separately, we should definitely output the same info as done by MutableTestExecutionSummary. I think all we'd need would be a TestExecutionListener that keeps hold of the TestPlan passed to testPlanExecutionStarted which we could use to access information about parent descriptors. Do you think you could give that a try?

1. Add a `testPlan` getter to `platform/launcher/listeners/SummaryGeneratingListener.java`. 2. DiffPrinter now needs testPlan to display the testMethod info. Issue: junit-team#3139

XJ114514 · 2024-10-19T17:05:44Z

If we list them separately, we should definitely output the same info as done by MutableTestExecutionSummary. I think all we'd need would be a TestExecutionListener that keeps hold of the TestPlan passed to testPlanExecutionStarted which we could use to access information about parent descriptors. Do you think you could give that a try?

The printFoundTestsSummary() was not called / MutableTestExecutionSummary was not created in ConsoleTestExecutor in summary mode. What's worse, the testPlan was never visible to ConsoleTestExecutor
Therefore, I add a testPlan getter to platform/launcher/listeners/SummaryGeneratingListener.java. Will that create any security issue?

Update DiffPrinter to contain testPlan as an attribute and display the testMethod info when printing. Preview is below.

marcphilipp

The printFoundTestsSummary() was not called / MutableTestExecutionSummary was not created in ConsoleTestExecutor in summary mode. What's worse, the testPlan was never visible to ConsoleTestExecutor
Therefore, I add a testPlan getter to platform/launcher/listeners/SummaryGeneratingListener.java. Will that create any security issue?

Rather than adding it to SummaryGeneratingListener, I'd register another (local) listener for this purpose.

Update DiffPrinter to contain testPlan as an attribute and display the testMethod info when printing. Preview is below.

Looks like we're getting closer. We're still missing the (2) etc. references, though.

XJ114514 · 2024-10-21T12:17:11Z

The printFoundTestsSummary() was not called / MutableTestExecutionSummary was not created in ConsoleTestExecutor in summary mode. What's worse, the testPlan was never visible to ConsoleTestExecutor
Therefore, I add a testPlan getter to platform/launcher/listeners/SummaryGeneratingListener.java. Will that create any security issue?

Rather than adding it to SummaryGeneratingListener, I'd register another (local) listener for this purpose.

Update DiffPrinter to contain testPlan as an attribute and display the testMethod info when printing. Preview is below.

Looks like we're getting closer. We're still missing the (2) etc. references, though.

I don't know how to create a local listener. Could you step in and make one? Sorry for my ignorance.
In CI/CD checks, many tests were ignored and were treated as failed tests in the report after I made the change, do you know what is wrong?
My console output has no (2) reference before each test failure.

marcphilipp · 2024-10-21T13:23:57Z

I don't know how to create a local listener. Could you step in and make one? Sorry for my ignorance.

For some reason, GitHub didn't let me push a commit to your PR so I opened a PR: XJ114514#1

In CI/CD checks, many tests were ignored and were treated as failed tests in the report after I made the change, do you know what is wrong?

We use Predictive Test Selection so the prior build that ignored them probably didn't select the ones that failed in the most recent build.

My console output has no (2) reference before each test failure.

Sorry for not being clear. I think we should add those there as well.

add testPlanCapturer listener

This reverts commit eef975b.

XJ114514 · 2024-10-25T12:38:12Z

For some reason, GitHub didn't let me push a commit to your PR so I opened a PR: XJ114514#1

I manually combined the change you made to this PR, but the testPlan from testPlanCapturer is null when using. Can you check about it, sorry I can not offer help because I don't understand how the TestExecutionListener works.

Sorry for not being clear. I think we should add those there as well.

I found the testIdentifier.getUniqueId() has the sequential numbers at the end and it works for my small test cases.
Would you use this standalone generated from current PR to run special cases to see if it will work? You can also generate the standalone by going to the branch and run gradle.
getUniqueId() code:

preview from my two simple test methods

XuJ321 and others added 13 commits August 31, 2024 18:25

Issue: junit-team#3139 part 1

e0d2849

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence To do list: 1. Add diff function dependency 2. Using diff to generate the output desired Issue: junit-team#3139

Issue: junit-team#3139 part 2

bc1b464

Tasks completed: 1. Locate ConsoleLauncher output 2. Check if actual and expected of AssertionFailedError are both type of CharSequence 3. Add diff function dependency 4. Using diff to generate the output desired Issue: junit-team#3139

Issue: junit-team#3139 part 2 update

2fa9a8e

delete one line of extra code

Merge branch 'junit-team:main' into main

e8d11bd

Fix format issue

7b4088d

fix build documentation issue

bb29695

trying to fix the build error

2c27b53

Try to Fix Dependency issue

af16685

Issue: junit-team#3139

Issue: junit-team#3139 Add external diff module to module documentation

3d73d1e

Add external diff module to module documentation For unknown reason, the test cases can not find the external module while the gradle can build normally. Issue: junit-team#3139

Merge branch 'junit-team:main' into main

c8a7b95

Remove module-info.java documentation for testing

53d884c

Run spotless and do final testing before draft PR

26bcaa4

marcphilipp linked an issue Sep 23, 2024 that may be closed by this pull request

ConsoleLauncher should output diff of expected and actual String values #3139

Open

marcphilipp marked this pull request as draft September 23, 2024 13:30

marcphilipp mentioned this pull request Sep 23, 2024

Add diff section to report of AssertionFailedError failures #3397

Closed

6 tasks

sbrannen added type: enhancement theme: reporting component: Platform labels Sep 23, 2024

sbrannen changed the title ~~Issue: junit-team#3139 Draft PR~~ Output diff of expected and actual values in ConsoleLauncher Sep 23, 2024

marcphilipp requested changes Sep 25, 2024

View reviewed changes

XJ114514 added 2 commits September 25, 2024 07:58

Update libs.versions.toml

3248752

Adding diffPrinter class

04f789a

1. Add diffPrinter class 2. replace diff in consoleTestExecutor with diffPrinter - will be deleted evetually 3. mark the location of adding diff in flatPrinter and verboseTreePrinter. Issue: junit-team#3139

marcphilipp requested changes Oct 8, 2024

View reviewed changes

XJ114514 and others added 2 commits October 8, 2024 22:12

Update junit-platform-console/src/main/java/org/junit/platform/consol…

983e090

…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>

Update junit-platform-console/src/main/java/org/junit/platform/consol…

eef49f1

…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>

XJ114514 and others added 2 commits October 8, 2024 22:12

Update junit-platform-console/src/main/java/org/junit/platform/consol…

bcd0608

…e/tasks/DiffPrinter.java Co-authored-by: Marc Philipp <[email protected]>

adjust the diffPrinter output format

cc69bfb

XJ114514 closed this Oct 9, 2024

XJ114514 reopened this Oct 9, 2024

Update the rule of inlineDiffByWord

705b4cf

Only if one word has space, show inlineDiffByWord.

update the format of printing diff

2383f3e

Issue: junit-team#3139 Update to display each test method info

82f723a

1. Add a `testPlan` getter to `platform/launcher/listeners/SummaryGeneratingListener.java`. 2. DiffPrinter now needs testPlan to display the testMethod info. Issue: junit-team#3139

marcphilipp requested changes Oct 21, 2024

View reviewed changes

Print Diffs (Markdown) only once when needed

536e9c7

apply spotless

f44f929

marcphilipp mentioned this pull request Oct 21, 2024

Capture TestPlan in separate listener XJ114514/junit5#1

Closed

XJ114514 added 3 commits October 25, 2024 07:51

Combine PR made by Marc from #1

eef975b

add testPlanCapturer listener

Revert "Combine PR made by Marc from #1"

de00ade

This reverts commit eef975b.

Issue: junit-team#3139 Add number reference to each test

a3f8f25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Output diff of expected and actual values in `ConsoleLauncher` #4017

Output diff of expected and actual values in `ConsoleLauncher` #4017

XJ114514 commented Sep 23, 2024 •

edited

Loading

marcphilipp commented Sep 23, 2024

XJ114514 commented Sep 24, 2024

marcphilipp commented Sep 24, 2024

XJ114514 commented Sep 24, 2024

marcphilipp left a comment

marcphilipp Sep 25, 2024

XJ114514 Sep 25, 2024

marcphilipp Sep 26, 2024

XJ114514 Sep 27, 2024 •

edited

Loading

marcphilipp Oct 2, 2024

XJ114514 Oct 7, 2024

marcphilipp left a comment

marcphilipp Oct 8, 2024

XJ114514 Oct 9, 2024

marcphilipp Oct 9, 2024

XJ114514 Oct 9, 2024

XJ114514 commented Oct 12, 2024

marcphilipp commented Oct 14, 2024

XJ114514 commented Oct 19, 2024

marcphilipp left a comment

XJ114514 commented Oct 21, 2024 •

edited

Loading

marcphilipp commented Oct 21, 2024

XJ114514 commented Oct 25, 2024 •

edited

Loading

Output diff of expected and actual values in ConsoleLauncher #4017

Are you sure you want to change the base?

Output diff of expected and actual values in ConsoleLauncher #4017

Conversation

XJ114514 commented Sep 23, 2024 • edited Loading

Overview

Current PR can successfully build and generate junit-platform-console-standalone-1.12.0-SNAPSHOT.jar containing diff outputs for two different Strings to the console.

Definition of Done

marcphilipp commented Sep 23, 2024

XJ114514 commented Sep 24, 2024

marcphilipp commented Sep 24, 2024

XJ114514 commented Sep 24, 2024

marcphilipp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XJ114514 Sep 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marcphilipp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XJ114514 commented Oct 12, 2024

marcphilipp commented Oct 14, 2024

XJ114514 commented Oct 19, 2024

marcphilipp left a comment

Choose a reason for hiding this comment

XJ114514 commented Oct 21, 2024 • edited Loading

marcphilipp commented Oct 21, 2024

XJ114514 commented Oct 25, 2024 • edited Loading

Output diff of expected and actual values in `ConsoleLauncher` #4017

Output diff of expected and actual values in `ConsoleLauncher` #4017

XJ114514 commented Sep 23, 2024 •

edited

Loading

Current PR can successfully build and generate `junit-platform-console-standalone-1.12.0-SNAPSHOT.jar` containing diff outputs for two different Strings to the console.

XJ114514 Sep 27, 2024 •

edited

Loading

XJ114514 commented Oct 21, 2024 •

edited

Loading

XJ114514 commented Oct 25, 2024 •

edited

Loading