-
Notifications
You must be signed in to change notification settings - Fork 6
Use_case_Topic_encryption
See Strimzi's topic encryption.
See Strimzi's topic encryption.
See TopicIds.
More recent versions of Fetch request use topic ids, rather than topic names.
This means the proxy would need to know the mapping of id to name in order to know whether to decrypt the message batches (and the key with which to do it).
But in deployment with multiple proxies, a client might have made its Metadata request via some other proxy instance, so the proxy instance handling Fetch might not have previously observed a Metadata response which included that topic id.
The simplest solution would be for the proxy to force the client to use an older version of Fetch by changing the ApiVersions response. But this is only really a sticking plaster, as topic ids are the future, and it would prevent clients from using the features enabled by newer versions of Fetch.
For a better solution, the proxy would need to make "its own" Metadata requests in order to handle these more recent versions of the Fetch RPC.
Encrypting messages (or batches) makes them slightly bigger. It is possible that this increase in size pushes the batch above the topic's max.message.bytes, meaning the batch is rejected (MESSAGE_TOO_LARGE).
The simple work around is to change the producer batch.size configuration to avoid this. But in the worse case (very small messages) the increase in size of encrypted messages is proportionally very large, meaning you'd need to use a very small batch size.
A complex alternative would be for the proxy to fragment a single too-large batch into multiple smaller batches. But further complications arise because one of those fragmented batches might succeed but another might fail to be appended, and it's impossible to convey this distinction to the client, which is unaware of the fragmentation applied by the proxy. EOS in the proxy would solve that, but it's rather a heavyweight solution.