Integration Name

Apache Tomcat [apache_tomcat]

Dataset Name

No response

Integration Version

1.5.1

Agent Version

8.13.4

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.1

OS Version and Architecture

Windows Server 2022

Software/API Version

9.0.86

Error Message

Processor dissect with tag fail-dissect_event_original in pipeline _simulate_pipeline failed with message: Unable to find match for dissect pattern: %{_tmp.sourceorusername} %{apache_tomcat.access.http.ident} %{apache_tomcat.access.http.useragent} [%{_tmp.timestamp}] "%{http.request.method} %{url.original} HTTP/%{http.version}" %{_tmp.dissectgrok} against source: ###.###.###.### - - [24/Oct/2024:14:18:49 +1100] "-" 400 - ###.###.###.### + 0.000 "-" "-" X-Forwarded-For="-"",
"Processor grok with tag fail-grok_parse_log_sourceoruser in pipeline _simulate_pipeline failed with message: field [_tmp] not present as part of path [_tmp.sourceorusername]",
"Processor grok with tag fail-grok_parse_log_dissectgrok in pipeline _simulate_pipeline failed with message: field [_tmp] not present as part of path [_tmp.dissectgrok]

Event Original

81.2.69.144 - - [24/Oct/2024:14:18:49 +1100] "-" 400 - 81.2.69.145 + 0.000 "-" "-" X-Forwarded-For="-"

What did you do?

Configured my tomcats logs to output as below per the documentation.

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
       prefix="localhost_access_log" suffix=".txt"
       pattern='%h %l %u %t "%r" %s %b %A %X %T "%{Referer}i" "%{User-Agent}i" X-Forwarded-For="%{X-Forwarded-For}i"' />

Configure the integration and push the policy to the agent.

What did you see?

Log entry:

###.###.###.### - - [24/Oct/2024:14:18:49 +1100] "-" 400 - ###.###.###.### + 0.000 "-" "-" X-Forwarded-For="-"

Event error.message:

Processor dissect with tag fail-dissect_event_original in pipeline _simulate_pipeline failed with message: Unable to find match for dissect pattern: %{_tmp.sourceorusername} %{apache_tomcat.access.http.ident} %{apache_tomcat.access.http.useragent} [%{_tmp.timestamp}] \"%{http.request.method} %{url.original} HTTP/%{http.version}\" %{_tmp.dissectgrok} against source: ###.###.###.### - - [24/Oct/2024:14:18:49 +1100] \"-\" 400 - ###.###.###.### + 0.000 \"-\" \"-\" X-Forwarded-For=\"-\"",
             "Processor grok with tag fail-grok_parse_log_sourceoruser in pipeline _simulate_pipeline failed with message: field [_tmp] not present as part of path [_tmp.sourceorusername]",
             "Processor grok with tag fail-grok_parse_log_dissectgrok in pipeline _simulate_pipeline failed with message: field [_tmp] not present as part of path [_tmp.dissectgrok]

What did you expect to see?

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2024-10-24T03:18:49.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "-",
                "ident": "-"
              },
              "response_time": 0,
              "ip": {
                "local": "###.###.###.###"
              },
              "connection_status": "+"
            }
          },
          "related": {
            "ip": [
              "###.###.###.###",
              "###.###.###.###"
            ]
          },
          "http": {
            "request": {
              "referrer": "-"
            },
            "response": {
              "status_code": 400
            }
          },
          "source": {
            "ip": "###.###.###.###"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "failure",
            "module": "apache_tomcat"
          },
          "user_agent": {
            "original": "-"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:05:00.065851947Z"
        }
      }
    }
  ]
}

Anything else?

I've fixed this issue here in a fork Personal Repo

The dissect needs to be split out to prevent this error:

Current:

  - dissect:
      field: event.original
      tag: 'dissect_event_original'
      pattern: '%{_tmp.sourceorusername} %{apache_tomcat.access.http.ident} %{apache_tomcat.access.http.useragent} [%{_tmp.timestamp}] "%{http.request.method} %{url.original} HTTP/%{http.version}" %{_tmp.dissectgrok}'
      on_failure:
        - append:
            field: error.message
            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'

Recommeneded:

  - dissect:
      field: event.original
      tag: 'dissect_event_original'
      pattern: '%{_tmp.sourceorusername} %{apache_tomcat.access.http.ident} %{apache_tomcat.access.http.useragent} [%{_tmp.timestamp}] "%{_tmp.dissect_request}" %{_tmp.dissectgrok}'
      on_failure:
        - append:
            field: error.message
            value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag fail-{{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'
  - dissect:
      if: _tmp.dissect_request != '-'
      field: _tmp.dissect_request
      pattern: '%{http.request.method} %{url.original} HTTP/%{http.version}'