Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[KafkaIO] Share size and offset estimators among ReadFromKafkaDoFn instances per PTransform #32928

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open

[KafkaIO] Share size and offset estimators among ReadFromKafkaDoFn instances per PTransform #32928

wants to merge 8 commits into from

Commits on Oct 25, 2024

  1. Explicitly update partitions and start/end offsets for relevant mock … 

    …consumers
    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    fa71cd6 View commit details
    Browse the repository at this point in the history

  2. Revert "Set backlog in gauge metric (apache#31137)"

    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    891f2d1 View commit details
    Browse the repository at this point in the history

  3. Revert "Add Backlog Metrics to Kafka Splittable DoFn Implementation (a… 

    …pache#31281)"

This reverts commit fd4368f.
    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    25e58b8 View commit details
    Browse the repository at this point in the history

  4. Call reportBacklog in nextBatch to report split metrics more often

    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    cf4817e View commit details
    Browse the repository at this point in the history

  5. Report SDF metrics for the active split/partition after processing a … 

    …record batch
    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    6a07d66 View commit details
    Browse the repository at this point in the history

  6. Remove unused property, assignment in finalize will not be observed

    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    45583dd View commit details
    Browse the repository at this point in the history

  7. Use KafkaSourceDescriptor as cache key and log entry

    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    496886c View commit details
    Browse the repository at this point in the history

  8. Share AvgRecordSize and KafkaLatestOffsetEstimator caches among DoFns

    @sjvanrossum
    sjvanrossum committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    7df359f View commit details
    Browse the repository at this point in the history