log_retention_days Parameter Not Applied to CloudWatch Logs Due to Inaccessibility of Record Data

[trojaond]

Bug Report

Describe the bug
The log_retention_days parameter is ignored because it cannot access record data via record_accessor. In contrast, log_stream_template and log_group_template can. Consequently, when a new log group is created, the retention setting defaults to "Never expire."

eg.: record:

{
  "log_retention": 7,
  "group_name": "/kubernetes/addons/fluentbit-test",
  "stream_name": "nginx"
}

fluentbit output configuration:

[OUTPUT]
    Name cloudwatch_logs
    Match kube.*
    log_group_name  fallback-group_name
    log_stream_prefix fallback-stream
    log_group_template $group_name.  ✅
    log_stream_template $stream_name   ✅
    auto_create_group true
    log_retention_days $log_retention ❌ (this does not work)

Although Fluent Bit successfully pushes logs to CloudWatch, it sets their retention to "Never expire," which can be problematic.

To Reproduce

• Create fluent bit configuration
• Configure cloudwatch_log output to set log_retention_days to value of record

Expected behavior
The log retention settings defined in Fluent Bit are accurately propagated to CloudWatch. This means that the specified retention period should be applied automatically, ensuring logs are kept only for the intended duration.

Your Environment

• Version used: v3.2.4
• Configuration: EKS + Fluentbit + Cloudwatch

Additional context
This feature is particularly useful because it allows you to configure the retention period based on specific Kubernetes namespaces, deployments, and more. Moreover, the legacy Go CloudWatch plugin offers this capability.