Home

Welcome to the design wiki!

Proxy Use Cases

There are a number of potential use cases which have been identified which could be solved through a proxy at the Kafka protocol level. The list below is neither ordered nor exhaustive. Each page describes the use case and the associated challenges in more detail.

1. Configuration Policy Enforcement For instance enforcing limits on the total number of partitions within a cluster. There are open questions whether this is strictly enforceable and whether or not we should try to do it in upstream Kafka before resorting to a proxy
2. Topic Encryption Enable certain topics to be encrypted to avoid SREs who have access to the underlying disks to be able to read the data.
3. Tar Pit Slow down malicious or badly behaved users. The ability to disconnect individual connections might allow us more options wrt reauthentication.
4. Multitenancy Enable topic and user isolation within a single Kafka cluster
5. Authentication Performing authentication in a proxy reduces load from the Kafka brokers
6. Schema validation Ensure that all messages on a given topic adhere to a particular schema
7. Audit logging Keep track of who created a topic, how much did they produce, etc. Getting client IP Addresses (not currently possible)