Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd: introduce zstd compression support in Fluent Bit #9830

Open
wants to merge 19 commits into
base: master
Choose a base branch
from

Conversation

edsiper
Copy link
Member

@edsiper edsiper commented Jan 14, 2025

The following PR introduces ZSTD compression support in Fluent Bit:

  • adds zstd v1.56 (bundled library)
  • new flb_zstd.c compress/uncompress interface
  • internal interface unit tests
  • opentelemetry output: add support for compress: zstd
  • opentelemetry input: add support to detect and uncompress zstd

Test: Fluent Bit --> OpenTelemetry Collector

bin/fluent-bit -i dummy -p dummy="{\"zstd\": \"ok\"}" -o opentelemetry -p port=4318 -p compress=zstd

opentelemetry-collector:

2025-01-13T18:53:30.506-0600	info	Logs	{"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 1}
2025-01-13T18:53:30.506-0600	info	ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2025-01-14 00:53:29.832229967 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Map({"zstd":"ok"})
Trace ID: 
Span ID: 
Flags: 0
	{"kind": "exporter", "data_type": "logs", "name": "debug"}

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

robskillington and others added 6 commits January 13, 2025 17:14
Signed-off-by: Eduardo Silva <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
This patch adds a new compression mechanism called 'zstd', this is
available through the 'compress' configuration property, in the configuration file
it can be enabled with:

  pipeline:
     outputs:
      - name: opentelemetry
        match: '*'
        compress: zstd

Signed-off-by: Eduardo Silva <[email protected]>
leonardo-albertovich and others added 7 commits January 17, 2025 17:51
Additionaly the explicitly set content-length was removed when
compressing the requests body was removed.

Signed-off-by: Leonardo Alminana <[email protected]>
Signed-off-by: Eduardo Silva <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants