[filebeat] Add support for `bulk_max_size` and `preset` #42312

khushijain21 · 2025-01-15T09:03:03Z

Proposed commit message

This PR is a follow up to #41849. It improves test scenarios. It also adds support for bulk_max_size and preset.

We will have to wait for next release of otel-collector to use compression_level parameter. But the default compression level used by elasticsearchexporter is set to 1. So we can go ahead and support preset paramater now

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related Issue
Part of https://github.com/elastic/opentelemetry-dev/issues/614
open-telemetry/opentelemetry-collector-contrib#37260

mergify · 2025-01-15T09:03:46Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @khushijain21? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2025-01-15T09:03:47Z

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

elasticmachine · 2025-01-15T16:58:59Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

khushijain21 · 2025-01-16T03:54:20Z

/test

libbeat/otelbeat/beatconverter/beatconverter.go

rdner · 2025-01-16T08:24:14Z

libbeat/outputs/elasticsearch/elasticsearch.go

-	if !cfg.HasField("bulk_max_size") {
-		if err := cfg.SetInt("bulk_max_size", -1, defaultBulkSize); err != nil {
-			return outputs.Fail(err)
-		}
-	}
-


Why change the ES output implementation?

I'm wondering the same. I believe the ES output must not change as part of this PR.

This is only setting bulk_max_size to a default value if not set, I moved it here, to keep it consistent.

https://github.com/elastic/beats/pull/42312/files#diff-8cfe4be190c94b04b508a5fae989f3955e3362dadc14f203190cb2317e8992e7R77-R78

rdner · 2025-01-16T08:24:43Z

libbeat/outputs/elasticsearch/testdata/basic-op.yml

Was it deleted intentionally?

yes, I refactored the test cases to take inputs as yaml strings instead of reading from a file. This helped reuse common parts of the cofe. This also makes sure that as more paramaters will be mapped, we only have to update it in one place.

rdner · 2025-01-16T08:24:51Z

libbeat/outputs/elasticsearch/testdata/basic.yml

Was it deleted intentionally?

rdner · 2025-01-16T08:28:41Z

libbeat/otelbeat/beatconverter/beatconverter_test.go

-		require.NoError(t, err)
+	})
+
+	t.Run("test local queue setting is promoted to global level", func(t *testing.T) {


I'm struggling to understand this test case.
Why do we have queue as a part of the output configuration and then move to the filebeatreceiver level? It should always be on the filebeatreceiver level, right?

I have the same question as @rdner, the queue configuration in Beats is not part of the output.

It is possible to set queue both under filebeatreceiver level and at output level. When set under output level we make sure to promote it to filbeatreceiver level. This is done to allow existing
behavior of specifying queue settings at the top level. Also, when presets are applied, it is first set under output level. This logic ensures that queue settings that come from both user or presets are promoted one level up.

Ref PR: #36788
Our documentations currently do not mention setting queue at output level. Opened an issue for the same #42310

belimawr · 2025-01-16T16:25:09Z

libbeat/otelbeat/beatconverter/beatconverter_test.go

-		require.NoError(t, err)
+	})
+
+	t.Run("test local queue setting is promoted to global level", func(t *testing.T) {


I have the same question as @rdner, the queue configuration in Beats is not part of the output.

libbeat/outputs/elasticsearch/config_otel.go

belimawr · 2025-01-16T16:34:07Z

libbeat/outputs/elasticsearch/config_otel_test.go

-		})
-	}
+	t.Run("basic config translation", func(t *testing.T) {
+		basicInput := `


That's an output configuration, I'm a bit confused why it's called basicInput. Maybe it can be called esOutputCfg.

belimawr · 2025-01-16T16:34:55Z

libbeat/outputs/elasticsearch/config_otel_test.go

+  X-Header-1: foo
+  X-Bar-Header: bar`
+
+		basicOp := `


This name here is also confusing me a bit. What does op stand for here, operation?

What about renaming it to otelOutputCfg?

Suggested change

basicOp := `

otelOutputCfg := `

The Op stood for output. I'll rename it to your suggestion, thanks

libbeat/outputs/elasticsearch/config_otel_test.go

belimawr · 2025-01-16T16:39:26Z

libbeat/outputs/elasticsearch/elasticsearch.go

-	if !cfg.HasField("bulk_max_size") {
-		if err := cfg.SetInt("bulk_max_size", -1, defaultBulkSize); err != nil {
-			return outputs.Fail(err)
-		}
-	}
-


I'm wondering the same. I believe the ES output must not change as part of this PR.

cmacknz · 2025-01-16T22:07:30Z

It looks like we are mapping existing parameters in Beats to similarly named parameters in the OTel collector. This only makes sense if we know they behave equivalent from a performance perspective.

The bulk_max_size translating to max_size_items looks like it is probably OK, did we test that it behaves exactly the same? https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/README.md#batching

The presets I am much less sure about and I don't think we should touch them without benchmarking them. The parameter values looking the same doesn't matter, the performance characteristics being the same or better is what matters. How are we measuring that?

Co-authored-by: Tiago Queiroz <[email protected]>

khushijain21 · 2025-01-20T15:13:44Z

The bulk_max_size translating to max_size_items looks like it is probably OK, did we test that it behaves exactly the same?

Yes. It was tested with setting bulk_max_size to 1 and sending 1000 log events. The behavior in both modes was similar, i.e all documents were indexed

The presets I am much less sure about and I don't think we should touch them without benchmarking them. The parameter values looking the same doesn't matter, the performance characteristics being the same or better is what matters. How are we measuring that?

All parameters that preset overrides are supported including bulk_max_size as part of this PR. Compression level is always set to 1 on es-exporter so we should be good to go. We can compare the performance impact of preset in a follow up PR using the benchmark work being done here https://ingest-metricstore.kb.us-central1.gcp.cloud.es.io:9243/app/dashboards#/view/88664583-01a2-48e5-80e6-fc93db54fc86?_g=h@9d8efc3

khushijain21 added 2 commits January 13, 2025 09:20

TODO comments implemented

54d79b3

add bulk max size

ab00ea8

khushijain21 requested a review from a team as a code owner January 15, 2025 09:03

khushijain21 requested review from belimawr and leehinman January 15, 2025 09:03

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 15, 2025

mergify bot assigned khushijain21 Jan 15, 2025

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Jan 15, 2025

khushijain21 marked this pull request as draft January 15, 2025 09:04

add support for preset

44e3aeb

khushijain21 changed the title ~~[filebeat] Add support for bulk_max_size~~ [filebeat] Add support for bulk_max_size and preset Jan 15, 2025

khushijain21 marked this pull request as ready for review January 15, 2025 12:39

Merge branch 'main' into otelfollow

96ee378

khushijain21 requested a review from rdner January 15, 2025 12:39

khushijain21 marked this pull request as draft January 15, 2025 12:49

khushijain21 marked this pull request as ready for review January 15, 2025 16:48

khushijain21 added 2 commits January 15, 2025 22:22

make update and lint

a5a6535

nit

2bfa2ef

khushijain21 added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Jan 15, 2025

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 15, 2025

keep compression level

83decfa

rdner reviewed Jan 16, 2025

View reviewed changes

belimawr reviewed Jan 16, 2025

View reviewed changes

khushijain21 and others added 2 commits January 20, 2025 19:24

Update libbeat/outputs/elasticsearch/config_otel.go

f1971c7

Co-authored-by: Tiago Queiroz <[email protected]>

comments addressed

26d31c1

khushijain21 added 2 commits January 20, 2025 19:35

add strings

1800387

Merge branch 'main' into otelfollow

cd3c8a8

khushijain21 requested review from rdner and belimawr January 20, 2025 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filebeat] Add support for `bulk_max_size` and `preset` #42312

[filebeat] Add support for `bulk_max_size` and `preset` #42312

khushijain21 commented Jan 15, 2025 •

edited

Loading

mergify bot commented Jan 15, 2025

mergify bot commented Jan 15, 2025

elasticmachine commented Jan 15, 2025

khushijain21 commented Jan 16, 2025

rdner Jan 16, 2025

belimawr Jan 16, 2025

khushijain21 Jan 20, 2025

rdner Jan 16, 2025

khushijain21 Jan 20, 2025 •

edited

Loading

rdner Jan 16, 2025

rdner Jan 16, 2025

belimawr Jan 16, 2025

khushijain21 Jan 20, 2025 •

edited

Loading

belimawr Jan 16, 2025

belimawr Jan 16, 2025

belimawr Jan 16, 2025

khushijain21 Jan 20, 2025 •

edited

Loading

belimawr Jan 16, 2025

cmacknz commented Jan 16, 2025 •

edited

Loading

khushijain21 commented Jan 20, 2025 •

edited

Loading

[filebeat] Add support for bulk_max_size and preset #42312

Are you sure you want to change the base?

[filebeat] Add support for bulk_max_size and preset #42312

Conversation

khushijain21 commented Jan 15, 2025 • edited Loading

Proposed commit message

Checklist

mergify bot commented Jan 15, 2025

mergify bot commented Jan 15, 2025

elasticmachine commented Jan 15, 2025

khushijain21 commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushijain21 Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushijain21 Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khushijain21 Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz commented Jan 16, 2025 • edited Loading

khushijain21 commented Jan 20, 2025 • edited Loading

[filebeat] Add support for `bulk_max_size` and `preset` #42312

[filebeat] Add support for `bulk_max_size` and `preset` #42312

khushijain21 commented Jan 15, 2025 •

edited

Loading

khushijain21 Jan 20, 2025 •

edited

Loading

khushijain21 Jan 20, 2025 •

edited

Loading

khushijain21 Jan 20, 2025 •

edited

Loading

cmacknz commented Jan 16, 2025 •

edited

Loading

khushijain21 commented Jan 20, 2025 •

edited

Loading