Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline. #11515

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

[Bug] Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline. #11515

wants to merge 2 commits into from
+63 −1

Conversation

BrendanNurmi
Copy link

Proposed commit message

Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline.

This change seeks to resolve defect #11514, where the dissect will fail in the event that a 400 bad request occurs by modify the ingest pipeline for Apache Tomcat localhost access logs.

The pipeline currently does not cater for a null (-) %r, First line of the request (method and request URI) see Tomcat 9 - Access Logging.

To resolve this, the %r will be split out as a whole and dissected again in the event it's not null (-).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Added 400 test case to Integration Log sample.
  • Updated pipeline within ECE and tested using log samples within integration tests.
  • Updated Test Cases and results to expected vaules.

How to test this PR locally

  • Cloned existing pipeline "logs-apache_tomcat.access.1.5.1" within ECE deployment and modified to add in dissects and tested using the following records:

Input Docs:

[
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 + 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.2\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 302 - 81.2.69.145 + 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.2\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 X 400 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 50 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 40 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"127.0.0.1, 127.0.0.3\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:18:58:17 +0530] \"POST /host-manager/images/asf-logo.svg HTTP/1.1\" 200 20486 81.2.69.145 - \"http://localhost:8080/host-manager/html\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36\" X-Forwarded-For=\"\""
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:19:01:17 +0530] \"GET /manager/status HTTP/1.1\" 200 4654 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36"
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - admin [02/Mar/2023:19:02:25 +0530] \"GET / HTTP/1.1\" 200 11235"
    }
  },
  {
    "_index": "index",
    "_id": "id",
    "_source": {
      "message": "81.2.69.144 - - [24/Oct/2024:14:18:49 +1100] \"-\" 400 - 81.2.69.145 + 0.000 \"-\" \"-\" X-Forwarded-For=\"-\""
    }
  }
]

Results:

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              },
              "response_time": 400,
              "ip": {
                "local": "81.2.69.145"
              },
              "connection_status": "+"
            }
          },
          "related": {
            "ip": [
              "81.2.69.144",
              "81.2.69.145",
              "127.0.0.1",
              "127.0.0.2"
            ]
          },
          "destination": {
            "bytes": 20486
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "client": {
            "ip": [
              "127.0.0.1",
              "127.0.0.2"
            ]
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "_value": null,
          "timestamp": "2024-10-24T23:59:02.240317575Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              },
              "response_time": 400,
              "ip": {
                "local": "81.2.69.145"
              },
              "connection_status": "+"
            }
          },
          "related": {
            "ip": [
              "81.2.69.144",
              "81.2.69.145",
              "127.0.0.1",
              "127.0.0.2"
            ]
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 302
            }
          },
          "client": {
            "ip": [
              "127.0.0.1",
              "127.0.0.2"
            ]
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "_value": null,
          "timestamp": "2024-10-24T23:59:02.240331968Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              },
              "response_time": 400,
              "connection_status": "X"
            }
          },
          "related": {
            "ip": [
              "81.2.69.144",
              "127.0.0.1"
            ]
          },
          "destination": {
            "bytes": 20486
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "client": {
            "ip": [
              "127.0.0.1"
            ]
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "_value": null,
          "timestamp": "2024-10-24T23:59:02.240334601Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              },
              "response_time": 50
            }
          },
          "related": {
            "ip": [
              "81.2.69.144"
            ]
          },
          "destination": {
            "bytes": 20486
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:59:02.240339761Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              },
              "response_time": 40,
              "ip": {
                "local": "81.2.69.145"
              }
            }
          },
          "related": {
            "ip": [
              "81.2.69.144",
              "81.2.69.145",
              "127.0.0.1",
              "127.0.0.3"
            ]
          },
          "destination": {
            "bytes": 20486
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "client": {
            "ip": [
              "127.0.0.1",
              "127.0.0.3"
            ]
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "_value": null,
          "timestamp": "2024-10-24T23:59:02.2403418Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              }
            }
          },
          "related": {
            "ip": [
              "81.2.69.144"
            ]
          },
          "destination": {
            "bytes": 20486
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:59:02.240343772Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:28:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              },
              "ip": {
                "local": "81.2.69.145"
              },
              "connection_status": "-"
            }
          },
          "related": {
            "ip": [
              "81.2.69.144",
              "81.2.69.145"
            ]
          },
          "destination": {
            "bytes": 20486
          },
          "http": {
            "request": {
              "method": "POST",
              "referrer": "http://localhost:8080/host-manager/html"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/host-manager/images/asf-logo.svg",
            "extension": "svg",
            "original": "/host-manager/images/asf-logo.svg"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:59:02.240345692Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:31:17.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              }
            }
          },
          "related": {
            "ip": [
              "81.2.69.144"
            ]
          },
          "destination": {
            "bytes": 4654
          },
          "http": {
            "request": {
              "method": "GET",
              "referrer": "-"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/manager/status",
            "original": "/manager/status"
          },
          "user_agent": {
            "name": "Chrome",
            "original": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",
            "os": {
              "name": "Windows",
              "version": "10",
              "full": "Windows 10"
            },
            "device": {
              "name": "Other"
            },
            "version": "109.0.0.0"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:59:02.240348732Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2023-03-02T13:32:25.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "admin",
                "ident": "-"
              }
            }
          },
          "related": {
            "ip": [
              "81.2.69.144"
            ]
          },
          "destination": {
            "bytes": 11235
          },
          "http": {
            "request": {
              "method": "GET"
            },
            "version": "1.1",
            "response": {
              "status_code": 200
            }
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "success",
            "module": "apache_tomcat"
          },
          "url": {
            "path": "/",
            "original": "/"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:59:02.240350585Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "id",
        "_source": {
          "@timestamp": "2024-10-24T03:18:49.000Z",
          "ecs": {
            "version": "8.7.0"
          },
          "apache_tomcat": {
            "access": {
              "http": {
                "useragent": "-",
                "ident": "-"
              },
              "response_time": 0,
              "ip": {
                "local": "81.2.69.145"
              },
              "connection_status": "+"
            }
          },
          "related": {
            "ip": [
              "81.2.69.144",
              "81.2.69.145"
            ]
          },
          "http": {
            "request": {
              "referrer": "-"
            },
            "response": {
              "status_code": 400
            }
          },
          "source": {
            "ip": "81.2.69.144"
          },
          "event": {
            "category": [
              "web"
            ],
            "type": [
              "access"
            ],
            "kind": "event",
            "outcome": "failure",
            "module": "apache_tomcat"
          },
          "user_agent": {
            "original": "-"
          }
        },
        "_ingest": {
          "timestamp": "2024-10-24T23:59:02.240352292Z"
        }
      }
    }
  ]
}

Related issues

@BrendanNurmi BrendanNurmi requested a review from a team as a code owner October 25, 2024 00:13
@cla-checker-service CLA Checker Service
Copy link

cla-checker-service bot commented Oct 25, 2024 •
edited
Loading

💚 CLA has been signed

@andrewkroh andrewkroh added bugfix Pull request that fixes a bug issue Integration:apache_tomcat Apache Tomcat needs CLA User must sign the Elastic Contributor License before review. Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] and removed needs CLA User must sign the Elastic Contributor License before review. labels Oct 25, 2024
@BrendanNurmi BrendanNurmi changed the title [Bug] Adding Test case for 400 in test-access.log updated injest pipeline t… [Bug] Fix 400 bad request in localhost_access.*.log causing dissect failure on pipeline. Oct 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bugfix Pull request that fixes a bug issue Integration:apache_tomcat Apache Tomcat Team:Obs-InfraObs Label for the Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Apache Tomcat]: 400 in logs causes dissect to fail.
2 participants
@BrendanNurmi @andrewkroh