Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[winlogbeat] Default to raw api #42275

Merged
merged 25 commits into from
Jan 20, 2025
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
396fc7d
Default to raw api
marc-gr Jan 9, 2025
585ff5d
Merge remote-tracking branch 'upstream/main' into winlogbeat-default-api
marc-gr Jan 9, 2025
97ed84f
Install sysmon event manifests
marc-gr Jan 13, 2025
724e32b
Merge remote-tracking branch 'upstream/main' into winlogbeat-default-api
marc-gr Jan 13, 2025
d10097e
Re-generate golden files
marc-gr Jan 13, 2025
37302c6
Remove unused renderer option
marc-gr Jan 13, 2025
74858cb
Fix tests
marc-gr Jan 13, 2025
d829c9a
Fix tests
marc-gr Jan 14, 2025
8a66d4e
Remove field from docs
marc-gr Jan 14, 2025
24fca7a
Fix doc default value
marc-gr Jan 14, 2025
e6cc25b
Make script compatible with older ps
marc-gr Jan 14, 2025
5cfdc9b
Print error
marc-gr Jan 14, 2025
3fd5e3f
Fix web request
marc-gr Jan 14, 2025
61cb7a0
Clean tests and Fallback to use latest available metadata if non found
marc-gr Jan 14, 2025
bb6b1fb
Fix security pipeline for new rendered events
marc-gr Jan 15, 2025
18c4d33
Regenerate ingest golden files
marc-gr Jan 15, 2025
04a8a25
:Merge remote-tracking branch 'upstream/main' into winlogbeat-default…
marc-gr Jan 15, 2025
15fae95
Fix test for win2016
marc-gr Jan 15, 2025
c4b452e
Merge branch 'main' into winlogbeat-default-api
marc-gr Jan 16, 2025
62ca5fd
fix github action
marc-gr Jan 16, 2025
6dd9cf9
Update winlogbeat/sys/wineventlog/metadata_store_test.go
marc-gr Jan 16, 2025
f2cab51
simplify sysmon script
marc-gr Jan 16, 2025
56d88e9
Add retries and simplify call
marc-gr Jan 20, 2025
27de210
Remove unused function
marc-gr Jan 20, 2025
70e7a42
Merge remote-tracking branch 'upstream/main' into winlogbeat-default-api
marc-gr Jan 20, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 93 additions & 0 deletions .buildkite/scripts/install_sysmon.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
$downloadUrl = "https://live.sysinternals.com/Sysmon64.exe"
$tempFolder = "$env:TEMP\SysmonDownload"
$sysmonPath = "$tempFolder\Sysmon64.exe"

function Retry()
{
param(
[Parameter(Mandatory=$true)][Action]$action,
[Parameter(Mandatory=$false)][int]$maxAttempts = 3
)

$attempts=1
$ErrorActionPreferenceToRestore = $ErrorActionPreference
$ErrorActionPreference = "Stop"

do
{
try
{
$action.Invoke();
break;
}
catch [Exception]
{
Write-Host $_.Exception.Message
}

# exponential backoff delay
$attempts++
if ($attempts -le $maxAttempts) {
Write-Host("Action failed. Waiting " + $retryDelaySeconds + " seconds before attempt " + $attempts + " of " + $maxAttempts + ".")
Start-Sleep 5
}
else {
$ErrorActionPreference = $ErrorActionPreferenceToRestore
Write-Error $_.Exception.Message
}
} while ($attempts -le $maxAttempts)
$ErrorActionPreference = $ErrorActionPreferenceToRestore
}

if (!(Test-Path $tempFolder)) {
New-Item -ItemType Directory -Path $tempFolder
}

$ProgressPreference = 'SilentlyContinue'
function ParseErrorForResponseBody($Error) {
if ($PSVersionTable.PSVersion.Major -lt 6) {
if ($Error.Exception.Response) {
$Reader = New-Object System.IO.StreamReader($Error.Exception.Response.GetResponseStream())
$Reader.BaseStream.Position = 0
$Reader.DiscardBufferedData()
$ResponseBody = $Reader.ReadToEnd()
if ($ResponseBody.StartsWith('{')) {
$ResponseBody = $ResponseBody | ConvertFrom-Json
}
return $ResponseBody
}
}
else {
return $Error.ErrorDetails.Message
}
}

$attempts=1
do
{
$attempts++
try {
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$result = Invoke-WebRequest -Uri $downloadUrl -OutFile $sysmonPath -UseBasicParsing
break
}
catch {
$resp = ParseErrorForResponseBody($_)
Write-Host "$resp"
if ($attempts -gt 5) {
exit 1
}
}
} while ($attempts -le 5)

Write-Host "Sysmon64.exe downloaded successfully."

if ($sysmonPath) {
Start-Process -FilePath $sysmonPath -ArgumentList "-m" -Wait

Write-Host "Sysmon event manifest installation completed."
} else {
Write-Host "Sysmon executable not found in the downloaded archive."
}

Remove-Item -Path $tempFolder -Force -Recurse
6 changes: 6 additions & 0 deletions .buildkite/x-pack/pipeline.xpack.winlogbeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ steps:
- label: ":windows: x-pack/winlogbeat Win 2019 Unit Tests"
key: "mandatory-win-2019-unit-tests"
command: |
.buildkite/scripts/install_sysmon.ps1
Set-Location -Path x-pack/winlogbeat
mage build unitTest
retry:
Expand All @@ -91,6 +92,7 @@ steps:

- label: ":windows: x-pack/winlogbeat: Win 2016 Unit Tests"
command: |
.buildkite/scripts/install_sysmon.ps1
Set-Location -Path x-pack/winlogbeat
mage build unitTest
key: "mandatory-win-2016-unit-tests"
Expand Down Expand Up @@ -118,6 +120,7 @@ steps:

- label: ":windows: x-pack/winlogbeat: Win 2022 Unit Tests"
command: |
.buildkite/scripts/install_sysmon.ps1
Set-Location -Path x-pack/winlogbeat
mage build unitTest
key: "mandatory-win-2022-unit-tests"
Expand Down Expand Up @@ -150,6 +153,7 @@ steps:
steps:
- label: ":windows: x-pack/winlogbeat: Win 10 Unit Tests"
command: |
.buildkite/scripts/install_sysmon.ps1
Set-Location -Path x-pack/winlogbeat
mage build unitTest
key: "extended-win-10-unit-tests"
Expand Down Expand Up @@ -177,6 +181,7 @@ steps:

- label: ":windows: x-pack/winlogbeat: Win 11 Unit Tests"
command: |
.buildkite/scripts/install_sysmon.ps1
Set-Location -Path x-pack/winlogbeat
mage build unitTest
key: "extended-win-11-unit-tests"
Expand Down Expand Up @@ -204,6 +209,7 @@ steps:

- label: ":windows: x-pack/winlogbeat: Win 2019 Unit Tests"
command: |
.buildkite/scripts/install_sysmon.ps1
Set-Location -Path x-pack/winlogbeat
mage build unitTest
key: "extended-win-2019-unit-tests"
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/check-dev-tools.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,10 @@ jobs:
- uses: actions/setup-go@v5
with:
go-version-file: .go-version
- name: Fix Code is not compatible with Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Run check/update
run: |
go install github.com/magefile/mage
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
*Winlogbeat*

- Add "event.category" and "event.type" to Sysmon module for EventIDs 8, 9, 19, 20, 27, 28, 255 {pull}35193[35193]
- Default to use raw api and delete older xml implementation. {pull}42275[42275]

*Functionbeat*

Expand Down
6 changes: 0 additions & 6 deletions dev-tools/ecs-migration.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1972,12 +1972,6 @@

# renames to match Windows Event Log naming

- from: type
to: winlog.api
alias: true
beat: winlogbeat
rename: false

Comment on lines -1975 to -1980
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Question]

Why is this not needed any more?

Copy link
Contributor Author

@marc-gr marc-gr Jan 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We previously had wineventlog and wineventlog-experimental api's. The experimental API has been recently GA'd as wineventlog-raw in 8.18. For 9.0.0 we want to remove the old wineventlog api as it is less performant and the old way of rendering the events can be still achieved through config while using the new wineventlog-raw api.

So we will have a single api moving forward, that is why we can remove the winlog.api field references.

note that this changes are not going to be backported

- from: log_name
to: winlog.channel
alias: true
Expand Down
62 changes: 4 additions & 58 deletions filebeat/docs/inputs/input-winlog.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Here is a sample configuration:
==== `batch_read_size`

The maximum number of event log records to read from the Windows API in a single
batch. The default batch size is 100. Most Windows versions return an error if
batch. The default batch size is 512. Most Windows versions return an error if
the value is larger than 1024. *{vista_and_newer}*

{beatname_uc} starts a goroutine (a lightweight thread) to read from each
Expand Down Expand Up @@ -181,40 +181,6 @@ IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735).
event_id: 4624, 4625, 4700-4800, -4735
--------------------------------------------------------------------------------

[WARNING]
=======================================
If you specify more than 22 query conditions (event IDs or event ID ranges), some
versions of Windows will prevent {beatname_uc} from reading the event log due to
limits in the query system. If this occurs a similar warning as shown below will
be logged by {beatname_uc}, and it will continue processing data from other event
logs.

`WARN EventLog[Application] Open() error. No events will be read from this
source. The specified query is invalid.`

In some cases, the limit may be lower than 22 conditions. For instance, using a
mixture of ranges and single event IDs, along with an additional parameter such
as `ignore older`, results in a limit of 21 conditions.

If you have more than 22 conditions, you can workaround this Windows limitation
by using a drop_event[drop-event] processor to do the filtering after
{beatname_uc} has received the events from Windows. The filter shown below is
equivalent to `event_id: 903, 1024, 4624` but can be expanded beyond 22
event IDs.

[source,yaml]
--------------------------------------------------------------------------------
- type: winlog
name: Security
processors:
- drop_event.when.not.or:
- equals.winlog.event_id: 903
- equals.winlog.event_id: 1024
- equals.winlog.event_id: 4624
--------------------------------------------------------------------------------

=======================================

[float]
==== `language`

Expand Down Expand Up @@ -350,6 +316,9 @@ Example:
include_xml: true
--------------------------------------------------------------------------------

* This can have a significant impact on performance that can vary depending
on your system specs.

[float]
==== `tags`

Expand Down Expand Up @@ -434,26 +403,3 @@ stopped. *{vista_and_newer}*

Setting `no_more_events` to `stop` is useful when reading from archived event
log files where you want to read the whole file then exit.

[float]
==== `api`

This selects the event log reader implementation that is used to read events
from the Windows APIs. You should only set this option when testing experimental
features. When the value is set to `wineventlog-experimental` {beatname_uc} will
replace the default event log reader with the **experimental** implementation.
We are evaluating this implementation to see if it can provide increased
performance and reduce CPU usage. *{vista_and_newer}*

[source,yaml]
--------------------------------------------------------------------------------
- type: winlog
name: ForwardedEvents
api: wineventlog-experimental
--------------------------------------------------------------------------------

There are a few notable differences in the events:

* Events that contained data under `winlog.user_data` will now have it under
`winlog.event_data`.
* Setting `include_xml: true` has no effect.
11 changes: 0 additions & 11 deletions winlogbeat/_meta/fields.common.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,6 @@
All fields specific to the Windows Event Log are defined here.
fields:

- name: api
required: true
description: >
The event log API type used to read the record. The possible values are
"wineventlog" for the Windows Event Log XML reader or "wineventlog-raw" for its
more performant implementation.

- name: activity_id
type: keyword
required: false
Expand Down Expand Up @@ -499,10 +492,6 @@
version of Winlogbeat. These are added to the index template when
`migration.6_to_7.enable: true` is set in the configuration.
fields:
- name: type
type: alias
path: winlog.api
migration: true

- name: activity_id
type: alias
Expand Down
19 changes: 0 additions & 19 deletions winlogbeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -15372,15 +15372,6 @@ Field aliases based on Winlogbeat 6.x that point to the fields for this version



*`type`*::
+
--
type: alias

alias to: winlog.api

--

*`activity_id`*::
+
--
Expand Down Expand Up @@ -16279,16 +16270,6 @@ All fields specific to the Windows Event Log are defined here.



*`winlog.api`*::
+
--
The event log API type used to read the record. The possible values are "wineventlog" for the Windows Event Log XML reader or "wineventlog-raw" for its more performant implementation.


required: True

--

*`winlog.activity_id`*::
+
--
Expand Down
23 changes: 4 additions & 19 deletions winlogbeat/docs/winlogbeat-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ winlogbeat.event_logs:
==== `event_logs.batch_read_size`

The maximum number of event log records to read from the Windows API in a single
batch. The default batch size is 100. Most Windows versions return an error if
batch. The default batch size is 512. Most Windows versions return an error if
the value is larger than 1024. *{vista_and_newer}*

{beatname_uc} starts a goroutine (a lightweight thread) to read from each
Expand Down Expand Up @@ -376,6 +376,9 @@ winlogbeat.event_logs:
include_xml: true
--------------------------------------------------------------------------------

* This can have a significant impact on performance that can vary depending
on your system specs.

[float]
==== `event_logs.tags`

Expand Down Expand Up @@ -462,24 +465,6 @@ Setting `no_more_events` to `stop` is useful when reading from archived event
log files where you want to read the whole file then exit. There's a complete
example of how to read from an `.evtx` file in the <<reading-from-evtx,FAQ>>.

[float]
==== `event_logs.api`

This selects the event log reader implementation that is used to read events
from the Windows APIs. When the value is set to `wineventlog-raw` Winlogbeat will
replace the default XML event log reader with a more performant implementation.
*{vista_and_newer}*

[source,yaml]
--------------------------------------------------------------------------------
winlogbeat.event_logs:
- name: ForwardedEvents
api: wineventlog-raw
--------------------------------------------------------------------------------

* If `include_xml` is `true` the performance will be the same as the default API,
as performance improvements are lost when parsing the XML.


[float]
==== `overwrite_pipelines`
Expand Down
Loading
Loading