[dvc][server] Support separate inc push topic with different pubsub entries #1262
+81
−13
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Background
In the effort of supporting separate inc push topic feature and persisting its state on disk, in addition, to minimize schema evolution and code changes, we decided to embed state of "separate inc push topic" into existing fields, e.g.
upstreamOffsetMap
ofPartitionState
. TheupstreamOffsetMap
currently has # of entries that correspond to the number of regions we've built. However the problem is the key of the map is the pubsub server url. Technically, in the same region, the pubsub server url is used for both RT and separate inc push topic so we needs to append a special suffix "_sep" as new keys to differentiate. This also implies a new cluster id, a new alias.Summary
As describe in the background, while we have new cluster id, new alias, and new pubsub server url for separate inc push topic, in some certain circumstances, it gotta be resolved to the
normal
form in order for pubsub subscription.This PR makes sure these values will be resolved as needed in the consumer thread and drainer thread.
I checked
kafka url
andcluster id
usage in every downstream method and confirmed only these places are needed to resolve from new values to the old ones:KafkaConsumerService
andTopicManager
How was this PR tested?
Pending new unit tests.
Does this PR introduce any user-facing changes?