Vector don't send logs from kubernetes cluster

[ZrytyADHD]

Hi,
I try to deploy Vector V0.44 on Tanzu Kubernetes Cluster.
I was able to do that using helm and application work also use this config below to test log sending to my external graylog:

[sources.demo_logs]
format = "syslog"
interval = 0.5
type = "demo_logs"

[sinks.graylog_syslog]
address = "graylog:12201"
inputs = ["demo_logs"]
mode = "tcp"
type = "socket"

[sinks.graylog_syslog.encoding]
codec = "gelf"

And everything seems to work I can se this demo logs.
Problem is with kubernetes logs.

With this config in vector.toml:

[sources.kubernetes_logs]
type = "kubernetes_logs"
auto_partial_merge = true
delay_deletion_ms = 50

fingerprint_lines = 1
glob_minimum_cooldown_ms = 60000
ingestion_timestamp_field = ".ingest_timestamp"
max_line_bytes = 64768
max_read_bytes = 22048
oldest_first = true
read_from = "beginning"
self_node_name = "${VECTOR_SELF_NODE_NAME}"
timezone = "local"

logs are not sended to graylog, I think because they're not collected.

Can anyone help me or provide correct setting to collect them and sending very long logs like complete java stack traces?
Tanzu use containerd. With other collector like fluentbit or fluentd logs work perfectly.

[pront]

Do you have any sample log? Do you see any errors? Based on your sink configuration, you are trying to encode the log events with the gelf codec.

[ZrytyADHD]

Hi,
thanks for reply. I using gelf codec because logs are sended to Graylog which use it as input.
If I configure demo logs they are sended without problem even logs with more than 70k characters.

The problem is when I try collect them from my kubernetes cluster.
Example log from Velero below:
time="2025-01-21T09:13:30Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:142" time="2025-01-21T09:13:30Z" level=info msg="BackupStorageLocations is valid, marking as available" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:127" time="2025-01-21T09:13:30Z" level=info msg="plugin process exited" backup-storage-location=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-storage-location id=234707 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws time="2025-01-21T09:14:06Z" level=info msg="plugin process exited" backupLocation=velero/default cmd=/plugins/velero-plugin-for-aws controller=backup-sync id=234720 logSource="pkg/plugin/clientmgmt/process/logrus_adapter.go:80" plugin=/plugins/velero-plugin-for-aws time="2025-01-21T09:14:30Z" level=info msg="Validating BackupStorageLocation" backup-storage-location=velero/default controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:142"

I can't see any errors in vector:

Container: vector
Connected
2025-01-21T21:06:12.373274Z  INFO vector::app: Log level is enabled. level="info"
2025-01-21T21:06:12.380710Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2025-01-21T21:06:12.384682Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="vector-0"
2025-01-21T21:06:12.400820Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Including matching files. ret=["**/*"]
2025-01-21T21:06:12.400924Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}: vector::sources::kubernetes_logs: Excluding matching files. ret=["**/*.gz", "**/*.tmp"]
2025-01-21T21:06:12.402014Z  INFO vector::topology::running: Running healthchecks.
2025-01-21T21:06:12.402187Z  INFO vector: Vector has started. debug="false" version="0.44.0" arch="x86_64" revision="3cdc7c3 2025-01-13 21:26:04.735691656"
2025-01-21T21:06:12.402209Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2025-01-21T21:06:12.404511Z  INFO source{component_kind="source" component_id=kube_log component_type=kubernetes_logs}:file_server: file_source::checkpointer: Attempting to read legacy checkpoint files.
2025-01-21T21:06:12.409574Z  INFO vector::topology::builder: Healthcheck passed.

In fluentbit I have this config:

 Name               tail
 Tag                kube.*
  Path               /var/log/containers/*.log
  Parser             docker
  DB                 /var/log/flb_graylog.db
    DB.Sync            Normal
    Docker_Mode        On
    Buffer_Chunk_Size  512KB
    Buffer_Max_Size    5M
    Rotate_Wait        30
    Mem_Buf_Limit      30MB
    Skip_Long_Lines    On
    Refresh_Interval   10

and everything work fine.

I choose to use vector because I need collect long logs from docker such as java starcktrace which fluentd and fluentbit can't concat together.

[pront]

So you can have plenty of options to help you diagnose here:

• vector tap to see outputs of your source (and hence what goes in to the sink) - guide here
• vector top to see sink metrics - docs here
• add an internal_metrics source and inspect the sink metrics

E.g.

sources:
  internal_metrics_1:
    type: internal_metrics
    scrape_interval_secs: 10

I would start by looking at https://vector.dev/docs/reference/configuration/sinks/socket/#component_sent_events_total and https://vector.dev/docs/reference/configuration/sinks/socket/#component_discarded_events_total.

Let me know if the above makes sense or if you have followup questions.

Tip: you can create blocks with three backticks (```)

[ZrytyADHD]

Hi,
yes there is any output and this make mi curious, even nightly release not working.
I try very different configs :
from basic only with:

[sources.kubernetes_logs]
  type = "kubernetes_logs"

also try specify namespace/log location etc.
Tanzu use photon OS, I also try K3s on Redhat.
Maybe the problem is with vector deployment config because on kubernetes it require ENV:

value: info
  - name: VECTOR_SELF_NODE_NAME

Below my deployment config which i use - it's installed from Helm but during trying to resolve problem I modify it a little:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vector
  annotations:
    meta.helm.sh/release-name: vector
    meta.helm.sh/release-namespace: fluentd
  generation: 96
  labels:
    app.kubernetes.io/component: Aggregator
    app.kubernetes.io/instance: vector
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vector
    app.kubernetes.io/version: 0.44.0-distroless-libc
    helm.sh/chart: vector-0.40.0
  namespace: fluentd
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: Aggregator
      app.kubernetes.io/instance: vector
      app.kubernetes.io/name: vector
  template:
    metadata:
      labels:
        app.kubernetes.io/component: Aggregator
        app.kubernetes.io/instance: vector
        app.kubernetes.io/name: vector
        vector.dev/exclude: 'true'
      annotations:
      creationTimestamp:
      namespace: fluentd
    spec:
      containers:
        - args:
            - '--config-dir'
            - /etc/vector/
          env:
            - name: VECTOR_LOG
              value: info
            - name: VECTOR_SELF_NODE_NAME
          image: timberio/vector:0.44.0-distroless-libc
          imagePullPolicy: IfNotPresent
          name: vector
          ports:
            - containerPort: 8282
              name: datadog-agent
              protocol: TCP
            - containerPort: 24224
              name: fluent
              protocol: TCP
            - containerPort: 5044
              name: logstash
              protocol: TCP
            - containerPort: 8080
              name: splunk-hec
              protocol: TCP
            - containerPort: 8125
              name: statsd
              protocol: TCP
            - containerPort: 9000
              name: syslog
              protocol: TCP
            - containerPort: 6000
              name: vector
              protocol: TCP
            - containerPort: 9090
              name: prom-exporter
              protocol: TCP
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /vector-data-dir
              name: data
            - mountPath: /etc/vector/
              name: config
              readOnly: true
            - mountPath: /var/log/pods
              name: var-log
              readOnly: true
          _init: false
          __active: true
          resources: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      serviceAccount: vector
      serviceAccountName: vector
      terminationGracePeriodSeconds: 60
      volumes:
        - hostPath:
            path: /var/log/
            type: Directory
          name: var-log
        - name: data
          emptyDir: {}
        - name: config
          projected:
            defaultMode: 420
            sources:
              - configMap:
                  name: vector-toml
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  serviceName: vector-headless
  updateStrategy:
    rollingUpdate:
      partition: 0

[pront]

This is a complex setup. I recommend eliminating as much parameters as possible. Maybe keep a simple kubernetes_logs source and a console sink and see if that is feeding the console with logs.

[ZrytyADHD]

Ok,
I have some new info.
First I changed type of deployment of vector from agregator to agent.
It deploy as fluentd in 4 pods.
Config map below:

data_dir: /vector-data-dir
api:
  enabled: true
  address: 127.0.0.1:8686
  playground: false
sources:
  kubernetes_logs:
    type: kubernetes_logs
sinks:
  graylog_syslog:
    address: x.x.x.x:12216
    inputs:
      - kubernetes_logs
    mode: tcp
    type: socket
    encoding:
      codec: gelf

The first change is the logs are collected:

Now i can observe new problem in logs:

2025-01-28T19:45:13.271872Z ERROR sink{component_kind="sink" component_id=graylog_syslog component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] has been suppressed 154 times.
2025-01-28T19:45:13.271876Z ERROR sink{component_kind="sink" component_id=graylog_syslog component_type=socket}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Failed serializing frame." internal_log_rate_limit=true
2025-01-28T19:45:13.271894Z ERROR sink{component_kind="sink" component_id=graylog_syslog component_type=socket}: vector::internal_events::codecs: Internal log [Failed serializing frame.] is being suppressed to avoid flooding.
2025-01-28T19:45:13.271904Z ERROR sink{component_kind="sink" component_id=graylog_syslog component_type=socket}: vector_common::internal_event::component_events_dropped: Internal log [Events dropped] is being suppressed to avoid flooding.

[pront]

This suggests Vector is unintentionally dropping events. The error indicates a framing/codec issue. The log events as they are coming out of the kubernetes_logs source cannot be encoded by the socket sink. You specified:

    encoding:
      codec: gelf

Please take a look at other framing options: https://vector.dev/docs/reference/configuration/sources/socket/#framing

[ZrytyADHD]

Hi,
first thanks for help and providing me some ideas.
I was able to resolve my problem changing deployment type.
Default when installing, Vector is acting as Aggregator, so I reinstall it as agent and it start working with this config:

data_dir: /vector-data-dir
api:
  enabled: true
  address: 127.0.0.1:8686
  playground: false

sources:
  kubernetes_logs:
    type: "kubernetes_logs" 
    auto_partial_merge: true

sinks:
  graylog:
    type: "socket"
    inputs: ["kubernetes_logs"]
    mode: tcp
    address: "graylog_server:12216"
    encoding:
      codec: json

healthchecks:
  enabled: true

also i have to change codec from gelf to json.
Even auto_partial_merge: true work like as charm with logs over 100k characters.

Now I working to remap some fields, and i want to ask how to change remap field visible in graylog as:
kubernetes_pod_node_name

to
kubernetes_host or kubernetes_node_name

for example.

Also i have a question how to preserve logs from apps to not get empty gaps when vaector is updated. In fluentbit when I do this always hgave some 2-3 blank spot.