Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streamline native injection w.r.t cloning #2001

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
Current
Fixed: GITHUB-1994: Prevent duplication of XmlTest objects when its used as a parameter for Native Injection (Krishnan Mahadevan)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: Typo in the summary for the addition related to the Assertion class.

- Added a method in the Assertion class to enable downstream TestNG consumers to override the error message (Ryan Laseter)
+ Added a method in the Assertion class to allow downstream TestNG consumers to override the error message (Ryan Laseter)

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
Fixed: GITHUB-1994: Prevent duplication of XmlTest objects when its used as a parameter for Native Injection (Krishnan Mahadevan)
Fixed: GITHUB-1994: Prevent duplication of XmlTest objects when its used as a parameter for Native Injection (Krishnan Mahadevan)
Added a method in the Assertion class to allow downstream TestNG consumers to override the error message (Ryan Laseter)

New: Added a method in Assertion class to allow downstream TestNG consumers to override the error message (Ryan Laseter)
Fixed: GITHUB-165: @AfterGroups is not executed when group member fails or is skipped (Krishnan Mahadevan)
Fixed: GITHUB-118: @BeforeGroups only called if group is specified explicitly (Krishnan Mahadevan)
Expand Down
16 changes: 16 additions & 0 deletions src/main/java/org/testng/annotations/SkipCloning.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package org.testng.annotations;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move it in the internal/annotations package

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@juherr - Here's my rationale behind leaving this exposed ( I also would like to evangelize about this lesser known behavior of TestNG )

Currently when it comes to parameter injection and associating that with the TestResult object, we resort to cloning() if the user's parameter object class implements the Cloneable interface. So there could be a chance wherein the parameters that the user is altering in their tests is perhaps not reflected in their listeners or in their reports because we cloned them.

I think we should enrich our documentation to first call out this behavior explicitly ( a most common use case is a data driven test that is run via a @DataProvider annotation) and via this newly introduced annotation, still let the user control whether they want or do not want their test objects to be cloned by TestNG.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree because you want to make public something which wasn't asked.
Then, the clone solution was just a quick workaround for a design issue when the same object is used in reports. But the clone solution doesn't cover all the need.

That's why I have some doubts to publish a workaround of a workaround before be really sure how we need to fix the root issue.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree because you want to make public something which wasn't asked.

The changeset that introduced setting cloned variants of parameters was to fix #447 It was done by you via commit 78001fd

Via this commit we ended up implicitly cloning a parameter (if it was clone able) but never made this information public. IMO we should have done this, because an end-user should know if the real parameter that is being passed to the test method is being used or if its a cloned one.

We haven't heard of anyone mention about this problem only because usually our users dont implement the Cloneable interface to the parameter that they are passing to a data provider.

The moment someone does that, this implicit behavior will become evident.

So now it looks like the root cause was us introducing the Cloneable parameter.
What if the user provided parameter was clone able but the user didn't want it to be clone able ? How would they be able to support that. With collections it may not be possible because a user cannot add that annotation to a collection class (I think this also should be possible if we just enrich @SkipCloning annotation such that it can be added to a parameter just as how a user adds @Optional), but at least with a user's test project defined objects it should be something that they should be able to do.

This was my thinking.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I understand your point of view. But mine is the #447 was just a not complete workaround of the root issue.
I agree to release another workaround to fix the first workaround but I'm not feeling it good to make it public because we will have to manage deprecation.
Instead, I prefer to make it private and to find a good solution for the root issue.
Furthermore, I'm not sure my old commit will still work on with Java9 modules due to clone.setAccessible(true).

For the current PR:

Copy link
Member Author

@krmahadevan krmahadevan Jan 9, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok.. how about this one.

For the current issue.

  1. Rename SkipCloning to Clone (This annotation will still be public)
  2. Revamp our logic to not look for Cloneable implementation alone, but also expect that the user annotate the parameter using @Clone annotation and only if both these are satisfied we resort to cloning. (This will break Issue with TextReporter output #447 but we are fixing it in the right way. All the user would need to do is add the newly created annotation as well to their Collection based parameters)
  3. Remove XmlTest from being annotated with any annotation.

For a permanent fix for #447 the following would be the recommendation (We can call this out in the documentation)

If TestNG should use a copy of the parameters then the following should be done

  1. Annotate the parameter using @Clone
  2. The parameter should implement Cloneable interface.

If for this also there's no agreement then I will leave this PR open and then start with the next issue (because I guess we would have come to consensus that Lets agree to disagree)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ping @juherr

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SkipCloning or Clone, the problem is still the same: I don't see the use case of the feature.
The Cloneable fix was a quick fix which should be think again.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without giving the control of deciding which parameter to be cloned and which not to, how do you propose the problem will be fixed ? Can you please help elaborate that ?

@cbeust what are you thoughts on this ?



import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.Target;

/**
* Marker interface which when used on a type will ensure that TestNG does not clone the object but
* instead uses it as is when TestNG resorts to dependency injection.
*/
@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE})
public @interface SkipCloning {

}
14 changes: 13 additions & 1 deletion src/main/java/org/testng/internal/TestResult.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import org.testng.Reporter;
import org.testng.TestNGException;
import org.testng.TestRunner;
import org.testng.annotations.SkipCloning;
import org.testng.collections.Lists;
import org.testng.collections.Objects;

Expand Down Expand Up @@ -276,7 +277,7 @@ public void setParameters(Object[] parameters) {
m_parameters = new Object[parameters.length];
for (int i = 0; i < parameters.length; i++) {
// Copy parameter if possible because user may change it later
if (parameters[i] instanceof Cloneable) {
if (canAttemptCloning(parameters[i]) ) {
try {
Method clone = parameters[i].getClass().getDeclaredMethod("clone");
m_parameters[i] = clone.invoke(parameters[i]);
Expand All @@ -292,6 +293,17 @@ public void setParameters(Object[] parameters) {
}
}

private static boolean canAttemptCloning(Object parameter) {
if (parameter == null) {
return false;
}
SkipCloning skipCloning = parameter.getClass().getAnnotation(SkipCloning.class);
if (skipCloning != null) {
return false;
}
return parameter instanceof Cloneable;
}

@Override
public Object getInstance() {
return IParameterInfo.embeddedInstance(this.m_method.getInstance());
Expand Down
30 changes: 30 additions & 0 deletions src/main/java/org/testng/xml/XmlTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,15 @@

import org.testng.TestNG;
import org.testng.TestNGException;
import org.testng.annotations.SkipCloning;
import org.testng.collections.Lists;
import org.testng.collections.Maps;
import org.testng.xml.dom.ParentSetter;

import static org.testng.xml.XmlSuite.ParallelMode.skipDeprecatedValues;

/** This class describes the tag &lt;test&gt; in testng.xml. */
@SkipCloning
public class XmlTest implements Cloneable {

public static final int DEFAULT_TIMEOUT_MS = Integer.MAX_VALUE;
Expand Down Expand Up @@ -75,6 +77,31 @@ private void init(XmlSuite suite, int index) {
// For YAML
public XmlTest() {}

/**
* This constructor acts as a copy constructor. Please note that it does not automatically
* associate the copied {@link XmlTest} object with the current {@link XmlSuite} object and
* requires it to be done explicitly.
*
* @param xmlTest - The {@link XmlTest} object to copy from.
*/
public XmlTest(XmlTest xmlTest) {
XmlTest result = new XmlTest();
result.setName(getName());
result.setIncludedGroups(getIncludedGroups());
result.setExcludedGroups(getExcludedGroups());
result.setJUnit(isJUnit());
result.setParallel(getParallel());
result.setVerbose(getVerbose());
result.setParameters(getLocalParameters());
result.setXmlPackages(getXmlPackages());
result.setTimeOut(getTimeOut());

Map<String, List<String>> metagroups = getMetaGroups();
for (Map.Entry<String, List<String>> group : metagroups.entrySet()) {
result.addMetaGroup(group.getKey(), group.getValue());
}
}
Comment on lines +80 to +103
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The copy constructor in XmlTest is implemented correctly but contains an unnecessary instantiation of XmlTest (XmlTest result = new XmlTest();) which is not used. Remove this line to clean up the constructor.

-    XmlTest result = new XmlTest();

Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
/**
* This constructor acts as a copy constructor. Please note that it does not automatically
* associate the copied {@link XmlTest} object with the current {@link XmlSuite} object and
* requires it to be done explicitly.
*
* @param xmlTest - The {@link XmlTest} object to copy from.
*/
public XmlTest(XmlTest xmlTest) {
XmlTest result = new XmlTest();
result.setName(getName());
result.setIncludedGroups(getIncludedGroups());
result.setExcludedGroups(getExcludedGroups());
result.setJUnit(isJUnit());
result.setParallel(getParallel());
result.setVerbose(getVerbose());
result.setParameters(getLocalParameters());
result.setXmlPackages(getXmlPackages());
result.setTimeOut(getTimeOut());
Map<String, List<String>> metagroups = getMetaGroups();
for (Map.Entry<String, List<String>> group : metagroups.entrySet()) {
result.addMetaGroup(group.getKey(), group.getValue());
}
}
/**
* This constructor acts as a copy constructor. Please note that it does not automatically
* associate the copied {@link XmlTest} object with the current {@link XmlSuite} object and
* requires it to be done explicitly.
*
* @param xmlTest - The {@link XmlTest} object to copy from.
*/
public XmlTest(XmlTest xmlTest) {
result.setName(getName());
result.setIncludedGroups(getIncludedGroups());
result.setExcludedGroups(getExcludedGroups());
result.setJUnit(isJUnit());
result.setParallel(getParallel());
result.setVerbose(getVerbose());
result.setParameters(getLocalParameters());
result.setXmlPackages(getXmlPackages());
result.setTimeOut(getTimeOut());
Map<String, List<String>> metagroups = getMetaGroups();
for (Map.Entry<String, List<String>> group : metagroups.entrySet()) {
result.addMetaGroup(group.getKey(), group.getValue());
}
}


public void setXmlPackages(List<XmlPackage> packages) {
m_xmlPackages = Lists.newArrayList(packages);
}
Expand Down Expand Up @@ -482,8 +509,11 @@ public String toXml(String indent) {
* <p>The &lt;classes&gt; sub element is ignored for the moment.
*
* @return a clone of the current XmlTest
* @Deprecated - This method stands deprecated as of TestNG 7.0.0. Please make use of
* {@link XmlTest#XmlTest(XmlTest)} instead.
*/
@Override
@Deprecated
public Object clone() {
XmlTest result = new XmlTest(getSuite());

Expand Down
10 changes: 9 additions & 1 deletion src/test/java/test/parameters/ParameterTest.java
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package test.parameters;

import org.testng.ITestNGListener;
import org.testng.ITestResult;
import org.testng.TestListenerAdapter;
import org.testng.TestNG;
Expand All @@ -15,6 +14,7 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import test.parameters.issue1994.TestclassSample;

import static org.assertj.core.api.Assertions.assertThat;

Expand Down Expand Up @@ -130,4 +130,12 @@ public void testNativeInjectionAndParamsInjection() {
testng.run();
assertThat(listener.getPassedTests().isEmpty()).isFalse();
}

@Test(description = "GITHUB-1994")
public void testToEnsureNativeInjectionDoesnotResortToCloning() {
XmlSuite xmlsuite = createXmlSuite("suite", "test", TestclassSample.class);
TestNG testng = create(xmlsuite);
testng.run();
assertThat(TestclassSample.count).isEqualTo(1);
}
}
32 changes: 32 additions & 0 deletions src/test/java/test/parameters/issue1994/TestclassSample.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
package test.parameters.issue1994;

import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Listeners;
import org.testng.annotations.Test;
import org.testng.xml.XmlTest;

@Listeners(TestclassSample.class)
public class TestclassSample implements ISuiteListener {

public static int count = 0;

@BeforeClass
public void beforeClass(XmlTest xmlTest) {
}
Comment on lines +15 to +17
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The beforeClass method is empty. If this is intentional, consider adding a comment explaining why; otherwise, implement the necessary logic or remove it if unused.


@Test
public void testMethod() {
}
Comment on lines +19 to +21
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The testMethod is empty. Similar to beforeClass, add a comment if this is intentional, implement required logic, or remove if unused.


@Override
public void onFinish(ISuite suite) {
setCount(suite.getXmlSuite().getTests().size());
}

private static void setCount(int count) {
TestclassSample.count = count;
}

}