Rewrite_tag not working as expected when we used Namespaced Custom Resource Definitions

[jkrbharath]

Describe the issue

I have been diligently following the documentation to establish Namespaced Custom Resource Definitions (CRDs) within our environment. However, during the setup process, I encountered an error message that reads:

"Created task=0x7f86a808f6a0 id=0 without routes, dropping."

This error suggests that Fluent Bit, the logging aggregator we're using, encountered a task without a defined route for processing. Despite following the setup instructions provided in the documentation, I'm uncertain about the root cause of this issue.

Context:

• Fluent Operator is deployed via Helm. Fluent Bit is integrated as our logging aggregator and configured using the following namespace label:

namespaceFluentBitCfgSelector:
  matchLabels:
    onboardedFluentbit: "true"

ClusterInput looks like

apiVersion: fluentbit.fluent.io/v1alpha2
 kind: ClusterInput
metadata:
  name: test
   labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/component: logging
spec:
  tail:
     tag: kube.var.log.container.test
     path: "/var/log/containers/*.log"
     excludePath: ""
     parser: cri
     refreshIntervalSeconds: 10
     memBufLimit: 100MB
     skipLongLines: true
     db: /fluent-bit/tail/pos.db
     dbSync: Normal

Fluentbit config for namespace app-1 looks like

apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  name: test
  namespace: app-1
   labels:
     onboardedFluentbit: "true"
spec:
   outputSelector:
     matchLabels:
       fluentbit.fluent.io/enabled: "true"
       app.kubernetes.io/name: "test"

output Crds looks like

apiVersion: fluentbit.fluent.io/v1alpha2
kind: Output
metadata:
  name: stdout
  namespace: app-1
  labels:
      fluentbit.fluent.io/enabled: "true"
      app.kubernetes.io/name: "test"
spec:
  match: kube.var.log.container.test
  stdout:
    format: msgpack

config which is stored in secret named fluent-bit-config looks like

[Service]
    Http_Server    true
    Parsers_File    parsers.conf
[Input]
    Name    tail
    Path    /var/log/containers/*.log
    Refresh_Interval    10
    Skip_Long_Lines    true
    DB    /fluent-bit/tail/pos.db
    DB.Sync    Normal
    Mem_Buf_Limit    100MB
    Parser    cri
    Tag    kube.var.log.container.test
[Filter]
    Name    kubernetes
    Match    kube.*
    Kube_URL    https://kubernetes.default.svc:443
    Kube_CA_File    /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File    /var/run/secrets/kubernetes.io/serviceaccount/token
    Labels    false
    Annotations    false
[Filter]
    Name    nest
    Match    kube.*
    Operation    lift
    Nested_under    kubernetes
    Add_prefix    kubernetes_
[Filter]
    Name    modify
    Match    kube.*
    Remove    stream
    Remove    kubernetes_pod_id
    Remove    kubernetes_host
    Remove    kubernetes_container_hash
[Filter]
    Name    nest
    Match    kube.*
    Operation    nest
    Wildcard    kubernetes_*
    Nest_under    kubernetes
    Remove_prefix    kubernetes_
[Filter]
    Name    rewrite_tag
    Match    kube.*
    Rule    $kubernetes['namespace_name'] ^(app-1)$ 5d24f63865268abd3e56d1e2b430162a.$TAG false
    Emitter_Name    re_emitted_5d24f63865268abd3e56d1e2b430162a
[Output]
    Name    stdout
    Match    5d24f63865268abd3e56d1e2b430162a.kube.var.log.container.test
    Format    msgpack

It got to do with rewrite_tag which is added after having namespaced crs.
it's probably not able to tell the dynamic matching works

Thanks

How did you install fluent operator?

Fluent Operator is deployed via Helm

Additional context

No response

[SvenThies]

Hello @jkrbharath,

can you provide more information? What is the release you use? How did you install it exactly via helm (cmd and/or values.yaml)?

Thx!

[jkrbharath]

Hi @SvenThies
Fluent Operator is deployed via Helm.
version: 2.7.0

[SvenThies]

Hey @jkrbharath,

sry for the late reply. Can you please also share the values.yaml you used? That will help to reproduce the error more efficiently.

Thx!

[RadOctocode]

I'm also experiencing this issue of rewrite tag for namespaced CRDs not working as expected. I have installed v3.1.0 via helm online. The rule Rule $kubernetes['namespace_name'] ^(argocd)$ c673f1e144a42bef9dab959465e5a0e8.$TAG false for rewrite_tag does not work because the logs do not automatically have the key kubernetes or namespace_name in the JSON of the logs. Is there documentation on how this is suppose to be defined/ how to set up the filters and parsers so that it is set?

[cw-Guo]

@RadOctocode you have to use kubernetes filter I guess.

[jkrbharath]

Hi @cw-Guo ,

Could you please provide some working examples of namespaced CRDs? Thanks!

[cw-Guo]

Hi @jkrbharath , I think the main issue here is the filters you use to processs the kubernetes related fields.

namespaced CR is just a practice how we can use fluent-bit, it relies on the $kubernetes['namespace_name'] fields to work.

So to be able to use it, you need the kubernetes filter to help you and avoid any processing over it in the cluster level. You can process it using namespaced CRs

[Filter]
    Name    rewrite_tag
    Match    kube.*
    Rule    $kubernetes['namespace_name'] ^(app-1)$ 5d24f63865268abd3e56d1e2b430162a.$TAG false
    Emitter_Name    re_emitted_5d24f63865268abd3e56d1e2b430162a

[rishwanriju]

We are using Helm for our deployment, we had no control over the rewrite tag. I would appreciate any reference example or documentation you can give me.

[rishwanriju]

Hi team,

We have updated our filter configuration to only use the Kubernetes filter. Here is the current filter configuration:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFilter
metadata:
    labels:
      fluentbit.fluent.io/component: logging
      fluentbit.fluent.io/enabled: "true"
    name: kubernetes
spec:
  filters:
    - kubernetes:
        annotations: false
        kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubeURL: https://kubernetes.default.svc:443
        kubeTagPrefix: kube.
        mergeLog: true
        mergeLogTrim: true
        keepLog: false
        k8sLoggingParser: true
        k8sLoggingExclude: false
        labels: true
        useKubelet: true
        bufferSize: 0k
        tlsVerify: false
        kubeletPort: 10250
        mergeLogKey: log_processed
    
  match: kube.*

Our output configuration is as follows:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: output
metadata:
  namespace: app-1
  annotations:
    meta.helm.sh/release-name: stdout
    meta.helm.sh/release-namespace: fluent
  labels:
    fluentbit.fluent.io/component: logging
    fluentbit.fluent.io/enabled: "true"
  name: stdout
spec:
  customPlugin:
    config: |
      Name   stdout
      Match  *

Additionally, we have our fluentbit configuration on the namespace level:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: FluentBitConfig
metadata:
  name: app-1-config
  namespace: app-1
  labels:
    onboardedFluentbit: "true"
spec:
  outputSelector:
    matchLabels:
      fluentbit.fluent.io/enabled: "true"
      app.kubernetes.io/name: "test"

Despite these configurations, we are still encountering the error "Created task=0x7f86a808f6a0 id=0 without routes, dropping." We are unsure about what we are missing. Your assistance in resolving this matter would be greatly appreciated.
Thank you.