[BUG] Document Level Monitor failing with ConcurrentModificationException

[jkrishnarao2003]

What is the bug?
I setup a Document Level Monitor in OpenSearch 2.10.0. It is failing to run with error ConcurrentModificationException.

How can one reproduce the bug?
Monitor Config

{
"name": "Monitor1",
"type": "monitor",
"monitor_type": "doc_level_monitor",
"enabled": true,
"schedule": {
"period": {
"unit": "MINUTES",
"interval": 1
}
},
"inputs": [
{
"doc_level_input": {
"description": "",
"indices": [
"rlr*"
],
"queries": [
{
"id": "e1a6c189-aece-47d3-8fe9-97cfc4b12d68",
"name": "query1",
"query": "message:"java.lang.OutOfMemoryError"",
"tags": [
"ERROR"
]
}
]
}
}
],
"triggers": [],
"ui_metadata": {
"schedule": {
"timezone": null,
"frequency": "interval",
"period": {
"unit": "MINUTES",
"interval": 1
},
"daily": 0,
"weekly": {
"tue": false,
"wed": false,
"thur": false,
"sat": false,
"fri": false,
"mon": false,
"sun": false
},
"monthly": {
"type": "day",
"day": 1
},
"cronExpression": "0 */1 * * *"
},
"monitor_type": "doc_level_monitor",
"doc_level_input": {
"queries": [
{
"id": "e1a6c189-aece-47d3-8fe9-97cfc4b12d68",
"queryName": "query1",
"field": "message",
"operator": "is",
"query": "java.lang.OutOfMemoryError",
"tags": [
"ERROR"
]
}
]
},
"search": {
"searchType": "graph"
}
}
}

Exception trace when starting the Monitor

[2024-05-27T22:35:10,135][ERROR][o.o.a.DocumentLevelMonitorRunner] [omhqp120d5.uprr.com] Failed running Document-level-monitor Monitor1
java.util.ConcurrentModificationException: null
at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:756) ~[?:?]
at java.util.LinkedHashMap$LinkedKeyIterator.next(LinkedHashMap.java:778) ~[?:?]
at org.opensearch.alerting.DocumentLevelMonitorRunner.runMonitor(DocumentLevelMonitorRunner.kt:141) [opensearch-alerting-2.10.0.0.jar:2.10.0.0]
at org.opensearch.alerting.DocumentLevelMonitorRunner$runMonitor$1.invokeSuspend(DocumentLevelMonitorRunner.kt) [opensearch-alerting-2.10.0.0.jar:2.10.0.0]
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
at kotlinx.coroutines.DispatchedTask.run(Dispatched.kt:233) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:594) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler.access$runSafely(CoroutineScheduler.kt:60) [kotlinx-coroutines-core-1.1.1.jar:?]
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:742) [kotlinx-coroutines-core-1.1.1.jar:?]

What is the expected behavior?
Is it a bug in OpenSearch or something wrong with my monitor config?

What is your host/environment?

• OS: CentOS
• Version 7
• Plugins: OpenSearch 2.10.0 opensearch-alerting plugin

[dblock]

Catch All Triage - 1 2 3 4 5 6