Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[winlogbeat] Default to raw api #42275

Merged
merged 25 commits into from
Jan 20, 2025
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
396fc7d
Default to raw api
marc-gr Jan 9, 2025
585ff5d
Merge remote-tracking branch 'upstream/main' into winlogbeat-default-api
marc-gr Jan 9, 2025
97ed84f
Install sysmon event manifests
marc-gr Jan 13, 2025
724e32b
Merge remote-tracking branch 'upstream/main' into winlogbeat-default-api
marc-gr Jan 13, 2025
d10097e
Re-generate golden files
marc-gr Jan 13, 2025
37302c6
Remove unused renderer option
marc-gr Jan 13, 2025
74858cb
Fix tests
marc-gr Jan 13, 2025
d829c9a
Fix tests
marc-gr Jan 14, 2025
8a66d4e
Remove field from docs
marc-gr Jan 14, 2025
24fca7a
Fix doc default value
marc-gr Jan 14, 2025
e6cc25b
Make script compatible with older ps
marc-gr Jan 14, 2025
5cfdc9b
Print error
marc-gr Jan 14, 2025
3fd5e3f
Fix web request
marc-gr Jan 14, 2025
61cb7a0
Clean tests and Fallback to use latest available metadata if non found
marc-gr Jan 14, 2025
bb6b1fb
Fix security pipeline for new rendered events
marc-gr Jan 15, 2025
18c4d33
Regenerate ingest golden files
marc-gr Jan 15, 2025
04a8a25
:Merge remote-tracking branch 'upstream/main' into winlogbeat-default…
marc-gr Jan 15, 2025
15fae95
Fix test for win2016
marc-gr Jan 15, 2025
c4b452e
Merge branch 'main' into winlogbeat-default-api
marc-gr Jan 16, 2025
62ca5fd
fix github action
marc-gr Jan 16, 2025
6dd9cf9
Update winlogbeat/sys/wineventlog/metadata_store_test.go
marc-gr Jan 16, 2025
f2cab51
simplify sysmon script
marc-gr Jan 16, 2025
56d88e9
Add retries and simplify call
marc-gr Jan 20, 2025
27de210
Remove unused function
marc-gr Jan 20, 2025
70e7a42
Merge remote-tracking branch 'upstream/main' into winlogbeat-default-api
marc-gr Jan 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
*Winlogbeat*

- Add "event.category" and "event.type" to Sysmon module for EventIDs 8, 9, 19, 20, 27, 28, 255 {pull}35193[35193]
- Default to use raw api and delete older xml implementation. {pull}42275[42275]

*Functionbeat*

Expand Down
60 changes: 3 additions & 57 deletions filebeat/docs/inputs/input-winlog.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -181,40 +181,6 @@ IDs to include (e.g. 4700-4800), and single event IDs to exclude (e.g. -4735).
event_id: 4624, 4625, 4700-4800, -4735
--------------------------------------------------------------------------------

[WARNING]
=======================================
If you specify more than 22 query conditions (event IDs or event ID ranges), some
versions of Windows will prevent {beatname_uc} from reading the event log due to
limits in the query system. If this occurs a similar warning as shown below will
be logged by {beatname_uc}, and it will continue processing data from other event
logs.

`WARN EventLog[Application] Open() error. No events will be read from this
source. The specified query is invalid.`

In some cases, the limit may be lower than 22 conditions. For instance, using a
mixture of ranges and single event IDs, along with an additional parameter such
as `ignore older`, results in a limit of 21 conditions.

If you have more than 22 conditions, you can workaround this Windows limitation
by using a drop_event[drop-event] processor to do the filtering after
{beatname_uc} has received the events from Windows. The filter shown below is
equivalent to `event_id: 903, 1024, 4624` but can be expanded beyond 22
event IDs.

[source,yaml]
--------------------------------------------------------------------------------
- type: winlog
name: Security
processors:
- drop_event.when.not.or:
- equals.winlog.event_id: 903
- equals.winlog.event_id: 1024
- equals.winlog.event_id: 4624
--------------------------------------------------------------------------------

=======================================

[float]
==== `language`

Expand Down Expand Up @@ -350,6 +316,9 @@ Example:
include_xml: true
--------------------------------------------------------------------------------

* This can have a significant impact on performance that can vary depending
on your system specs.

[float]
==== `tags`

Expand Down Expand Up @@ -434,26 +403,3 @@ stopped. *{vista_and_newer}*

Setting `no_more_events` to `stop` is useful when reading from archived event
log files where you want to read the whole file then exit.

[float]
==== `api`

This selects the event log reader implementation that is used to read events
from the Windows APIs. You should only set this option when testing experimental
features. When the value is set to `wineventlog-experimental` {beatname_uc} will
replace the default event log reader with the **experimental** implementation.
We are evaluating this implementation to see if it can provide increased
performance and reduce CPU usage. *{vista_and_newer}*

[source,yaml]
--------------------------------------------------------------------------------
- type: winlog
name: ForwardedEvents
api: wineventlog-experimental
--------------------------------------------------------------------------------

There are a few notable differences in the events:

* Events that contained data under `winlog.user_data` will now have it under
`winlog.event_data`.
* Setting `include_xml: true` has no effect.
21 changes: 3 additions & 18 deletions winlogbeat/docs/winlogbeat-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,9 @@ winlogbeat.event_logs:
include_xml: true
--------------------------------------------------------------------------------

* This can have a significant impact on performance that can vary depending
on your system specs.

[float]
==== `event_logs.tags`

Expand Down Expand Up @@ -462,24 +465,6 @@ Setting `no_more_events` to `stop` is useful when reading from archived event
log files where you want to read the whole file then exit. There's a complete
example of how to read from an `.evtx` file in the <<reading-from-evtx,FAQ>>.

[float]
==== `event_logs.api`

This selects the event log reader implementation that is used to read events
from the Windows APIs. When the value is set to `wineventlog-raw` Winlogbeat will
replace the default XML event log reader with a more performant implementation.
*{vista_and_newer}*

[source,yaml]
--------------------------------------------------------------------------------
winlogbeat.event_logs:
- name: ForwardedEvents
api: wineventlog-raw
--------------------------------------------------------------------------------

* If `include_xml` is `true` the performance will be the same as the default API,
as performance improvements are lost when parsing the XML.


[float]
==== `overwrite_pipelines`
Expand Down
22 changes: 9 additions & 13 deletions winlogbeat/eventlog/bench_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,21 +58,17 @@ func TestBenchmarkRead(t *testing.T) {
safeWriteEvent(t, writer, uint32(rand.Int63()%1000), strconv.Itoa(i)+" "+randomSentence(256))
}

for _, api := range []string{winEventLogAPIName, winEventLogExpAPIName} {
t.Run("api="+api, func(t *testing.T) {
for _, includexml := range []bool{true, false} {
for _, batchSize := range []int{10, 100, 500, 1000} {
t.Run(fmt.Sprintf("include_xml=%v/batch_size=%d", includexml, batchSize), func(t *testing.T) {
result := testing.Benchmark(benchmarkEventLog(api, includexml, batchSize))
outputBenchmarkResults(t, result)
})
}
}
})
for _, includexml := range []bool{true, false} {
for _, batchSize := range []int{10, 100, 500, 1000} {
t.Run(fmt.Sprintf("include_xml=%v/batch_size=%d", includexml, batchSize), func(t *testing.T) {
result := testing.Benchmark(benchmarkEventLog(includexml, batchSize))
outputBenchmarkResults(t, result)
})
}
}
}

func benchmarkEventLog(api string, includexml bool, batchSize int) func(b *testing.B) {
func benchmarkEventLog(includexml bool, batchSize int) func(b *testing.B) {
return func(b *testing.B) {
conf := mapstr.M{
"name": providerName,
Expand All @@ -81,7 +77,7 @@ func benchmarkEventLog(api string, includexml bool, batchSize int) func(b *testi
"include_xml": includexml,
}

log := openLog(b, api, nil, conf)
log := openLog(b, nil, conf)
defer log.Close()

events := 0
Expand Down
164 changes: 0 additions & 164 deletions winlogbeat/eventlog/cache.go

This file was deleted.

Loading
Loading