Disable Buffer for Kafka Source to Clickhouse Sink

[seilerre]

I got a Vector config where I am writing from kafka to clickhouse. As Kafka is supposed to persist events and hand them to the consumer (Vector) in a reasonable pace, I would like to disable the buffer in Vector, so that when Vector is too slow to consumer a consumer lag is build in Kafka. Is this a bad idea? In my Clickhouse sink I tried to set:

        buffer:
          max_events: 0
          type: memory

This however produces a config error.

[jszwedko]

I think you'd be interested in Vector's end-to-end acknowledgement feature. Enabling that would have the kafka source only commit offsets once the data has been sent downstream.