Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[compat][server][client][test] Global RT DIV improvement (part 2): Ch… #1257

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Commits on Oct 23, 2024

  1. [compat][server][client][test] Global RT DIV improvement (part 2): Ch…

    …unking support for DIV message
    
    This change mainly focuses on adding chunking support for DIV messages when they are produced to Kafka topics,
    as the size of DIV message can be large. We leverage today's chunking mechanism for regular records and extend
    it to support DIV with the following modifications:
    
    1. All the DIV messages are of type {@link MessageType#CONTROL_MESSAGE_DIV} in its KafkaKey and their corresponding
       KafkaMessageEnvelope uses Put as the payload.
    2. Inside the Put payload, the actual message is stored in the putValue field and the schemaId can have the following cases:
       - If the DIV message is non-chunked, the schemaId is set to GLOBAL_DIV_STATE.
       - If the DIV message is chunk message, the schemaId is set to CHUNK.
       - If the DIV message is a chunk manifest message, the schemaId is set to CHUNKED_VALUE_MANIFEST.
    3. ChunkAssembler is adapted, on the receiver side, to buffer, assemble, and deserialize DIV messages (chunked/non-chunked).
    lluwm committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    20381c9 View commit details
    Browse the repository at this point in the history