Add LogPatternTool

[qianheng-aws]

Description

Add LogPatternTool to support generate log patterns based on an input DSL.

Background of log pattern

Log pattern is a feature in Observability. It aims to find the common feature among retrieved logs.

The implementation of the log pattern feature in Observability includes 3 steps:

1. execute a PPL to retrieve the first log
2. set default pattern field, which is implemented in front-end. The default pattern field is found by scanning over the values of the first sample log. The field with the longest length of value will be the default pattern field. (Code Ref; For alert analysis, it seems meaningless to find pattern on the field with the longest length of value.)
3. execute a PPL to retrieve data and fill in log pattern table. Assuming field message is the pattern field, an example of such an PPL is:
   source=opensearch_dashboards_sample_data_flights | where timestamp >= ’[2024-09-09 16](tel:2024090916):16:30.321000' and timestamp <= '2024-09-10 07:16:30.322000' | patterns Dest | stats count(), take(Dest, 1) by patterns_field
   By default, the operator pattern in PPL will generate a new field named pattern_field which will ignore characters conatined by [a-zA-Z\d] in the original value(Code Ref; For alert analysis, it also seems less helpful to analysis on such pattern).

However, if customers use DSL to retrieve logs, it doesn't support such feature. Thus we need this tool to achieve this goal.

Implementation of LogPatternTool

This tool supports generating log patterns on the input dsl and index. It's implemented by
several steps:

1. Retrival [[${DOC_SIZE_FIELD}]] logs from index
2. Extract patterns for each retrieved log
   2.1 Find Pattern Field: If users provide parameter [[${PATTERN_FIELD}]], use it as the pattern
   field; Otherwise, find the string field with the longest length on the first log.
   2.2 Extract Pattern: If users provide parameter [[${PATTERN}]], compile it as a pattern;
   Otherwise, use [[${DEFAULT_IGNORED_CHARS}]]. It will remove all chars matching the pattern.
3. Group logs by their extracted patterns.
4. Find top N patterns with the largest sample log size.
5. For each found top N patterns, return [[${SAMPLE_LOG_SIZE}]] sample logs.

Example

POST _plugins/_ml/agents/V4pYKJIBf_90oTl_gmnp/_execute
{
    "parameters": {
       "selected_tools": ["LogPatternTool"],
       "top_n_pattern": "3",
       "sample_log_size": "20",
       "doc_size": 100,
        "input":  """
{"query":{"bool":{"filter":[{"range":{"timestamp":{"from":"1727172421713||-1h","to":"1727172421713","include_lower":true,"include_upper":true,"boost":1}}},{"term":{"response":{"value":"200","boost":1}}}],"adjust_pure_negative":true,"boost":1}}}
        """
    }
}

Related Issues

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[xluo-aws]

Should we stop if longestField is fewer than maybe 30 chars or continue to search for 10 more records in case the first one has outlier data?

[qianheng-aws]

It just imitates the logic of Discover log pattern.

It sounds good to avoid outliner, but it could also make the logic more complex and inconsistent with Discover.

[xluo-aws]

The above suggestion is not ideal, just want to trigger some discussion. It's ok to leave it as it is if we don't have better idea.

[yuye-aws]

You can implement the ut and it after resolving my comments

[yuye-aws]

Are docSize, topNPattern and sampleLogSize tool parameters or run time parameters?

[qianheng-aws]

run time parameters I think. But I think users can set them as well when creating this tool, and it should support override them in runtime.

[yuye-aws]

Did not see the PPL, but it makes sense to use DSL for search. Would you like to check whether the code can be reused with search index tool? In search index tool, it has specialized operations on connector, model and model_group index.

[qianheng-aws]

Did not see the PPL, but it makes sense to use DSL for search. Would you like to check whether the code can be reused with search index tool? In search index tool, it has specialized operations on connector, model and model_group index.

LogPatternTool aims to generate log pattern for DSL. PPL already supports pattern syntax itself.

[yuye-aws]

Good job @qianheng-aws ! You can go on with ut and it after resolving these two mistakes.

[zane-neo]

It's better to add ITs to cover the code, is there any concern on adding ITs?

[codecov]

Codecov Report

Attention: Patch coverage is 40.98361% with 72 lines in your changes missing coverage. Please review.

Project coverage is 77.75%. Comparing base (2b76e3c) to head (6b71d21).
Report is 25 commits behind head on main.

Files with missing lines	Patch %	Lines
...ava/org/opensearch/agent/tools/LogPatternTool.java	41.52%	62 Missing and 7 partials ⚠️
src/main/java/org/opensearch/agent/ToolPlugin.java	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #413      +/-   ##
============================================
- Coverage     81.78%   77.75%   -4.04%     
- Complexity      193      260      +67     
============================================
  Files            11       15       +4     
  Lines           961     1380     +419     
  Branches        137      196      +59     
============================================
+ Hits            786     1073     +287     
- Misses          121      231     +110     
- Partials         54       76      +22     

Flag	Coverage Δ
	77.75% <40.98%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[qianheng-aws]

It's better to add ITs to cover the code, is there any concern on adding ITs?

I'm going to add one for this PR.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/skills/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/skills/backport-2.x
# Create a new branch
git switch --create backport/backport-413-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7d726125bc9704d5a1fa01831f44c7c0d7b4aab5
# Push it to GitHub
git push --set-upstream origin backport/backport-413-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/skills/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-413-to-2.x.

[yuye-aws]

@qianheng-aws Please raise another PR with UT and IT.

[opensearch-trigger-bot]

The backport to 2.17 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/skills/backport-2.17 2.17
# Navigate to the new working tree
pushd ../.worktrees/skills/backport-2.17
# Create a new branch
git switch --create backport/backport-413-to-2.17
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7d726125bc9704d5a1fa01831f44c7c0d7b4aab5
# Push it to GitHub
git push --set-upstream origin backport/backport-413-to-2.17
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/skills/backport-2.17

Then, create a pull request where the base branch is 2.17 and the compare/head branch is backport/backport-413-to-2.17.