You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
assignedPartitions += p ->AssignedPartition(newMarkersQueue(endOffset -1), redeliverActor, None, None)
Case Scenario:
For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic.
The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2.
Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.
Problem:
SM3 would not be tracked by any redelivery tracker.
Possible Solution:
Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.
The text was updated successfully, but these errors were encountered:
The disableRedeliveryBefore argument is there to enable redeliveries only after all known data has been loaded from Kafka to build the MarkersQueue state on startup or after a restart.
It doesn't affect which markers are tracked or not. So in your example, SM3 would still be tracked and considered for redelivery (provided that there's some movement on the topic, and more markers are seen).
The setting only affects when markers will be checked for redeliveries.
Ok, I agree SM3 would be tracked but it would not be redelivered as redelivery would be disabled for that offset. This should not be the required behavior as redelivery is required to be done, as it was not even done by the first instance of redelivery tracker.
Just wanted to know the use of the disableRedeliveryBefore argument of the marker queue?
And as per the logic, I feel the following case scenario would be a failure.
for code reference :
kmq/core/src/main/scala/com.softwaremill.kmq/redelivery/ConsumeMarkersActor.scala
Line 71 in 391f9f9
Case Scenario:
For a Topic(X) there is a marker topic(MT). One redelivery tracker is listening to the marker topic.
The order of markers on a partition are SM1, SM2, EM1, EM2, where SMx is start-marker for some key x and EMx is end-marker for some key x. the redelivery tracker commits on Kafka till EM2.
Then redelivery tracker disconnects and a new redelivery consumer is created, and in between a new SM3 is published. But as the consumer connects after SM3, endOffset/ disabledRedeliveryBefore for MarkerQueue of partition is after the SM3 offset.
Problem:
SM3 would not be tracked by any redelivery tracker.
Possible Solution:
Rather than endOffset, the redelivery tracker can disable redelivery before the last committed offset.
The text was updated successfully, but these errors were encountered: