Xdump msg_filter ignored for Exceptions with null messages

[paulcheeseman]

Here's a simple testcase that throws and catches an IOException with a null Exception message:

import java.io.IOException;

public class ThrowIOException {
	public static void main(String[] args) {
		System.out.println("Test which throws a java.io.IOException");
		try {
			throw new java.io.IOException();
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

When you run this testcase with the following Xdump option:

-Xdump:stack:events=throw+systhrow,filter=java/io/IOException,msg_filter=testmessage

the agent is triggered:

> java "-Xdump:stack:events=throw+systhrow,filter=java/io/IOException,msg_filter=testmessage" ThrowIOException
Test which throws a java.io.IOException
JVMDUMP039I Processing dump event "throw", detail "java/io/IOException" at 2023/01/10 09:45:40 - please wait.
Thread=main (000001BF746559E0) Status=Running
        at ThrowIOException.main([Ljava/lang/String;)V (ThrowIOException.java:7)
JVMDUMP013I Processed dump event "throw", detail "java/io/IOException".
java.io.IOException
        at ThrowIOException.main(ThrowIOException.java:7)

This is unexpected - i.e. I would not expect a null Exception message to match the msg_filter.

If the testcase is modified to add an Exception message to the IOException the msg_filter, e.g.:

throw new java.io.IOException("foo");

the test works as expected - i.e. it doesn't match, and the agent is not triggered.

It seems that msg_filter has always behaved this way, but I have somehow never noticed. I think the problem is in trigger.c, starting here, where the return code (retCode) is set to J9RAS_DUMP_MATCH if the filter matches (i.e. the Exception name matches the string in the Xdump agent's filter):

openj9/runtime/rasdump/trigger.c

Lines 429 to 431 in cc263cc

	if (filter && parseWildcard(filter, strlen(filter), &needleString, &needleLength, &matchFlag) == 0) {
	if (wildcardMatch(matchFlag, needleString, needleLength, message, nbytes)) {
	retCode = J9RAS_DUMP_MATCH;

The code then checks the subfilter (i.e. the msg_filter), and it will only set retCode to J9RAS_DUMP_NO_MATCH if the Exception has a message. If it doesn't have a message, the code falls through and returns J9RAS_DUMP_MATCH:

openj9/runtime/rasdump/trigger.c

Lines 451 to 461 in cc263cc

	if (NULL != emessage) {
	buf = vmThread->javaVM->internalVMFunctions->copyStringToUTF8WithMemAlloc(vmThread, emessage, J9_STR_NULL_TERMINATE_RESULT, "", 0, stackBuffer, 256, &buflen);
	if (NULL != buf) {
	if (wildcardMatch(matchFlag, needleString, needleLength, buf, buflen)) {
	retCode = J9RAS_DUMP_MATCH;
	} else {
	retCode = J9RAS_DUMP_NO_MATCH;
	}
	}
	}

I think the code needs an else block for the if (NULL != emessage) conditional to set retCode to J9RAS_DUMP_NO_MATCH when the Exception message is null, i.e. here:

openj9/runtime/rasdump/trigger.c

Lines 461 to 462 in cc263cc

	}

[keithc-ca]

On the surface, it seems reasonable that if msg_filter=whatever is specified then an exception without that message (perhaps because it's null) should not trigger the related dump action. However, changing the behavior to be "reasonable" is just that - a change in behavior. Can anyone foresee a reasonable scenario where the current behavior (message is null or matches) would be desirable?

[smithwil]

During the time I've worked on support cases, I have not had a situation where I needed to capture dumps from exceptions with null detail message OR a message pattern, and it's difficult to imagine how that would come up. On the other hand, it has tripped me up multiple times when msg_filter did not seem to work, because a dump was taken when I thought it should be filtered out. I usually specify the range option at the same time to limit the number of dumps two one or two (to limit customer impact), and if all the dumps are 'used up' for exceptions with null message, we don't get any dump from the later exception with the matching message string.

[manqingl]

The intention of users using event "msg_filter" is that users only want to see the exceptions with the specified messages. It seems an exception with "null" exception message is not what users are asking for. I do not see this as a behavior change, but a correction of wrong behavior. Assuming this issue has been there since its beginning, it is indeed questionable why this issue has never been raised.

Is there any way to get exceptions which only have a null message? Or those which exclude the null message ones? Suggest to clarify it in the infoCenter.

[flavell]

Can anyone foresee a reasonable scenario where the current behavior (message is null or matches) would be desirable?
The Xdump documentation https://www.ibm.com/docs/en/sdk-java-technology/8?topic=options-xdump
promotes the msg_filter as a way to reduce the number of dump files produced.
The documentation does not say that msg_filter will not filter null/empty.
Is this really a requirement for -Xdump, msg_filer and null/empty Strings ?
If this undocumented behaviour is wanted then another -Xdump option can be added to trap the null/empty message.
.
On the other hand, it has tripped me up multiple times when msg_filter did not seem to work, because a dump was taken when I thought it should be filtered out.
Yes, msg_filter not filtering on null/empty is a behaviour that we are used to seeing but we, to the best of my knowledge, haven't diagnosed and attempted to address it.
This is a Serviceability issue.
Think back to all those times when Customers provided us with loads of dumps that we didn't want, complained because their spool had filled up, ...
This fix will save time and so money (looking through loads of documentation that we didn't want) and improve Customer satisfaction.

[paulcheeseman]

Is there any way to get exceptions which only have a null message? Or those which exclude the null message ones? Suggest to clarify it in the infoCenter.

If the problem described in the original report is fixed, it should be possible to exclude null message Exceptions by simply putting a * in the msg_filter.

I don't think there's a way to trigger only on Exceptions with a null message, and my suggested fix wouldn't change that. This would require something new, for example a special token in the msg_filter (e.g. %null). This would require some careful thought, and personally I don't think it's nearly as important as the main issue. I don't think I've ever encountered a need to only trigger on null message Exceptions.

[paulcheeseman]

I have just been hit by this again while working on a customer issue, and its preventing me from generating the dump I need. In this latest situation I don't know which method is throwing the Exception so I can't filter on a stack frame. I've tried filtering on the Exception message but I keep getting dumps triggered by Exceptions of the same type but with null messages.

I have some ideas about how to proceed but it may take a couple more rounds of data collection to get around this issue.

Has there been any progress towards a fix?

[paulcheeseman]

Thanks @keithc-ca !