KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft #18196

m1a2st · 2024-12-15T11:35:02Z

Jira: https://issues.apache.org/jira/browse/KAFKA-18225

we don't implement the ClientQuotaCallback#updateClusterMetadata in Kraft mode. We will implement it in 4.0 version to pass CustomQuotaCallbackTest test in Kraft mode.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala

TaiJuWu · 2024-12-15T12:57:23Z

core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala

@@ -565,3 +531,97 @@ class KRaftMetadataCache(
  }
 }

+object KRaftMetadataCache {
+
+  def toCluster(clusterId: String, image: MetadataImage): Cluster = {


This result is different with KraftMetadataCache#toCluster (the old will filter partitions base on listener).
If we decide to change this one, maybe we need to document it.

chia7712 · 2025-01-08T09:20:22Z

@m1a2st please fix the conflicts

# Conflicts: # core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

m1a2st · 2025-01-08T11:43:54Z

Thanks for @chia7712 reminder, resolve the conflict

chia7712

@m1a2st thanks for this fix!

chia7712 · 2025-01-09T16:28:11Z

core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

 import scala.collection.mutable.ArrayBuffer
 import scala.jdk.CollectionConverters._

-@Disabled("KAFKA-18213")


nice to see this test gets fixed. Could you please close KAFKA-18213 as duplicate?

I move it to L104, I didn't fix it

I will take a look in this fail test

chia7712 · 2025-01-09T16:31:53Z

core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala

@@ -48,6 +51,11 @@ class DynamicClientQuotaPublisher(
  ): Unit = {
    val deltaName = s"MetadataDelta up to ${newImage.highestOffsetAndEpoch().offset}"
    try {
+        val clientQuotaCallback = conf.getConfiguredInstance(QuotaConfig.CLIENT_QUOTA_CALLBACK_CLASS_CONFIG, classOf[ClientQuotaCallback])


we should use the callback in quotaManagers rather than creating a new one! They are different instances and so this approach can update the callback used by quotaManagers

chia7712 · 2025-01-09T16:34:44Z

metadata/src/main/java/org/apache/kafka/metadata/BrokerRegistration.java

@@ -235,6 +235,13 @@ public Optional<Node> node(String listenerName) {
        }
        return Optional.of(new Node(id, endpoint.host(), endpoint.port(), rack.orElse(null), fenced));
    }
+
+    public List<Node> nodes() {
+        List<Node> nodes = new ArrayList<>();


we can leverage existent method. for example:

return listeners.keySet().stream().flatMap(l -> node(l).stream()).toList();

chia7712

@m1a2st thanks for this patch. I test your patch on my local and please fix following code also:

deleteTopic needs to use authorized admin
You must call removeQuotaOverrides after creating group1_user2. Otherwise, the removeQuota method of the custom callback will not be invoked. This is an interesting discrepancy. In ZooKeeper mode, removeQuota is called when altering SASL/SCRAM credentials. However, in Kraft mode, this behavior is absent. I'm uncertain whether this constitutes a breaking change. It appears to be an unusual behavior, as it is triggered by the addition of users. Instead of implementing this peculiar behavior, I suggest updating the documentation of the callback to reflect the actual implementation.
cc @dajac and @cmccabe

chia7712 · 2025-01-13T10:24:16Z

core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

@@ -179,13 +212,12 @@ class CustomQuotaCallbackTest extends IntegrationTestHarness with SaslSetup {
  }

  private def createTopic(topic: String, numPartitions: Int, leader: Int): Unit = {
-    // TODO createTopic 
+    TestUtils.createTopicWithAdmin(createAdminClient(), topic, brokers, controllerServers, numPartitions)


You have to honor the leader to make replica leader hosted by correct broker - otherwise, this test will get flaky in the future

core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala

m1a2st · 2025-01-13T16:00:57Z

Thanks for @chia7712 review, addressed all comments

chia7712

@m1a2st thanks for this patch.

chia7712 · 2025-01-15T13:53:52Z

core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

@@ -337,6 +371,7 @@ object GroupedUserQuotaCallback {
  val DefaultProduceQuotaProp = "default.produce.quota"
  val DefaultFetchQuotaProp = "default.fetch.quota"
  val UnlimitedQuotaMetricTags = new util.HashMap[String, String]
+  val updateClusterMetadataCalls = new AtomicInteger


why to add this count?

chia7712 · 2025-01-15T14:03:11Z

core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala

+      quotaManagers.clientQuotaCallback().ifPresent(clientQuotaCallback => {
+        if (delta.topicsDelta() != null || delta.clusterDelta() != null) {
+          val cluster = KRaftMetadataCache.toCluster(clusterId, newImage)
+          clientQuotaCallback.updateClusterMetadata(cluster)


I don't think this line is required.

chia7712 · 2025-01-15T14:04:21Z

core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

@@ -136,9 +148,11 @@ class CustomQuotaCallbackTest extends IntegrationTestHarness with SaslSetup {
    // Create large number of partitions on another broker, should result in throttling on first partition
    val largeTopic = "group1_largeTopic"
    createTopic(largeTopic, numPartitions = 99, leader = 0)
+    user.removeThrottleMetrics()


this is unnecessary if we remove the unnecessary call of updateClusterMetadata

chia7712 · 2025-01-15T14:08:58Z

core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

    user.waitForQuotaUpdate(8000, 2500, defaultRequestQuota)
    user.produceConsume(expectProduceThrottle = true, expectConsumeThrottle = true)

+    user.removeQuotaOverrides()


Could we call removeQuotaOverrides after creating new user for consistency?

chia7712 · 2025-01-15T21:04:21Z

@m1a2st could you please revise the docs of ClientQuotaCallback? there are some stale documents.

chia7712 · 2025-01-18T01:37:38Z

@m1a2st could you please fix the conflicts?

# Conflicts: # core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

chia7712 · 2025-01-19T14:48:21Z

@m1a2st please fix the conflicts :(

# Conflicts: # core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

chia7712 · 2025-01-20T12:12:55Z

@m1a2st please fix the conflicts

# Conflicts: # core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala

chia7712 · 2025-01-20T12:27:20Z

core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala

-        Option(delta.clientQuotasDelta()).foreach { clientQuotasDelta =>
-          clientQuotaMetadataManager.update(clientQuotasDelta)
+      quotaManagers.clientQuotaCallback().ifPresent(clientQuotaCallback => {
+        if (delta.topicsDelta() != null || delta.clusterDelta() != null) {


How about moving this callback trigger to a individual publisher? That can decouple the callback and other publisher. Also, that can avoid producing incorrect error message?

# Conflicts: # server-common/src/main/java/org/apache/kafka/server/config/QuotaConfig.java

chia7712

@m1a2st thanks for this patch

chia7712 · 2025-01-23T16:42:59Z

core/src/main/scala/kafka/server/metadata/DynamicTopicClusterQuotaPublisher.scala

+import org.apache.kafka.image.loader.LoaderManifest
+import org.apache.kafka.server.fault.FaultHandler
+
+class DynamicTopicClusterQuotaPublisher (


Please add comments to this new publisher. we should emphasize this approach is temporary and there is a follow-up jira

chia7712 · 2025-01-23T16:50:42Z

core/src/main/scala/kafka/server/ControllerServer.scala

+      ))
+
+      // Set up the DynamicTopicClusterQuotaPublisher. This will enable quotas for the cluster and topics.
+      metadataPublishers.add(new DynamicTopicClusterQuotaPublisher(


Please add test for controller

chia7712 · 2025-01-23T16:52:17Z

core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala

@@ -545,3 +511,96 @@ class KRaftMetadataCache(
  }
 }

+object KRaftMetadataCache {


#18632 is trying to remove reference of KRaftMetadataCache, so maybe we can move the helpers to MetadataCache?

TaiJuWu · 2025-01-24T05:50:50Z

core/src/main/scala/kafka/server/ControllerServer.scala

+        clusterId,
+        config,
+        sharedServer.metadataPublishingFaultHandler,
+        "broker",


Controller?

m1a2st added 3 commits December 15, 2024 15:47

wip

5fe6d1a

add clusterId to DynamicClientQuotaPublisher

b3edea5

Draft a version

15d7dbb

github-actions bot added the triage PRs from the community label Dec 15, 2024

m1a2st changed the title ~~KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft~~ [WIP]KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft Dec 15, 2024

github-actions bot added core Kafka Broker kraft labels Dec 15, 2024

TaiJuWu reviewed Dec 15, 2024

View reviewed changes

core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala Outdated Show resolved Hide resolved

TaiJuWu reviewed Dec 15, 2024

View reviewed changes

mumrah added the ci-approved label Dec 15, 2024

m1a2st added 4 commits December 17, 2024 00:12

revert test

4123c70

Merge branch 'trunk' into KAFKA-18225

765d7f1

add new test

513b381

update the test

74b1006

m1a2st changed the title ~~[WIP]KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft~~ KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft Dec 17, 2024

Merge branch 'trunk' into KAFKA-18225

d1cfe97

github-actions bot removed the triage PRs from the community label Dec 19, 2024

Merge branch 'trunk' into KAFKA-18225

a4758e7

m1a2st added 2 commits January 8, 2025 19:39

Merge branch 'trunk' into KAFKA-18225

dee9a65

# Conflicts: # core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

resolve conflict

cd853b6

chia7712 reviewed Jan 9, 2025

View reviewed changes

m1a2st added 5 commits January 10, 2025 01:08

Merge branch 'trunk' into KAFKA-18225

48a8b41

addressed by comments

6bf41a7

addressed by comments

3fe1d7c

Merge branch 'trunk' into KAFKA-18225

ed50ea8

fix var name

f87db91

chia7712 reviewed Jan 13, 2025

View reviewed changes

update delete topic

57fad78

m1a2st added 2 commits January 13, 2025 23:28

addressed by comment

909d8f9

update when updateClusterMetadata success

5e88ee1

chia7712 reviewed Jan 15, 2025

View reviewed changes

addressed by comment

b3365f0

m1a2st added 3 commits January 16, 2025 08:31

Merge branch 'trunk' into KAFKA-18225

8e36e2f

update the doc

977a603

addressed the comment

de16d0e

github-actions bot added the clients label Jan 16, 2025

Merge branch 'trunk' into KAFKA-18225

9f04283

# Conflicts: # core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

m1a2st added 2 commits January 19, 2025 22:50

Merge branch 'trunk' into KAFKA-18225

830c99c

# Conflicts: # core/src/test/scala/integration/kafka/api/CustomQuotaCallbackTest.scala

fix the test

3a3e409

Merge branch 'trunk' into KAFKA-18225

e48ca88

# Conflicts: # core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala

chia7712 reviewed Jan 20, 2025

View reviewed changes

m1a2st added 6 commits January 20, 2025 20:37

fix the conflict

6f679f1

separator logic

be432e6

add license

af4cee5

revert unused change

39f542e

revert unused change

0bab1c7

Merge branch 'trunk' into KAFKA-18225

1ea0067

# Conflicts: # server-common/src/main/java/org/apache/kafka/server/config/QuotaConfig.java

chia7712 reviewed Jan 23, 2025

View reviewed changes

TaiJuWu reviewed Jan 24, 2025

View reviewed changes

ijuma requested a review from cmccabe January 24, 2025 06:31

m1a2st added 3 commits January 24, 2025 19:51

wip

929e87a

addressed by comments

101cbdc

add new test

9c28db2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft #18196

KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft #18196

m1a2st commented Dec 15, 2024

TaiJuWu Dec 15, 2024 •

edited

Loading

chia7712 commented Jan 8, 2025

m1a2st commented Jan 8, 2025

chia7712 left a comment

chia7712 Jan 9, 2025

m1a2st Jan 9, 2025

m1a2st Jan 9, 2025

chia7712 Jan 9, 2025

chia7712 Jan 9, 2025

chia7712 left a comment

chia7712 Jan 13, 2025

m1a2st commented Jan 13, 2025

chia7712 left a comment

chia7712 Jan 15, 2025

chia7712 Jan 15, 2025

chia7712 Jan 15, 2025

chia7712 Jan 15, 2025

chia7712 commented Jan 15, 2025

chia7712 commented Jan 18, 2025

chia7712 commented Jan 19, 2025

chia7712 commented Jan 20, 2025

chia7712 Jan 20, 2025

chia7712 left a comment

chia7712 Jan 23, 2025

chia7712 Jan 23, 2025

chia7712 Jan 23, 2025

TaiJuWu Jan 24, 2025

KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft #18196

Are you sure you want to change the base?

KAFKA-18225: ClientQuotaCallback#updateClusterMetadata is unsupported by kraft #18196

Conversation

m1a2st commented Dec 15, 2024

Committer Checklist (excluded from commit message)

TaiJuWu Dec 15, 2024 • edited Loading

Choose a reason for hiding this comment

chia7712 commented Jan 8, 2025

m1a2st commented Jan 8, 2025

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

m1a2st commented Jan 13, 2025

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chia7712 commented Jan 15, 2025

chia7712 commented Jan 18, 2025

chia7712 commented Jan 19, 2025

chia7712 commented Jan 20, 2025

Choose a reason for hiding this comment

chia7712 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TaiJuWu Dec 15, 2024 •

edited

Loading