Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When using KafkaBackend, the application ignores messages with an offset higher than 75/100 #552

Open
yoniv87 opened this issue Nov 8, 2022 · 0 comments · May be fixed by #553
Open

When using KafkaBackend, the application ignores messages with an offset higher than 75/100 #552

yoniv87 opened this issue Nov 8, 2022 · 0 comments · May be fixed by #553
Labels
bug Something isn't working

Comments

@yoniv87
Copy link

yoniv87 commented Nov 8, 2022

Describe the bug
When using KafkaBackend as the topology backend, in case there are more than 50 messages in the backend topic, the application ignores the state represented in the up-to-date messages in the topic.

To Reproduce
Steps to reproduce the behavior:

  1. Configure a backend topic for JulieOps with one partition only, and retention time -1 (infinite) or compacted topic
  2. Run multiple topologies with different topology keys (at least 100 times - to have 100 messages in the state topic)
  3. Try to run one of the topologies that were recently updated (e.g., the one that was updated on the 100th message)
  4. The application reads only the first 50/75 messages from the state/backend topic and ignores the latest state as it is described in the latest messages with a higher offset. The number of consumed messages can be between 45-75 messages per bulk.

Expected behavior
The application should read all messages before deciding on the current state to act according to the latest state.

Runtime (please complete the following information):

  • OS: Windows, Alpine, CentOS
  • JVM version: openjdk 11, java 17
  • Version 4.4.1

Additional context
In the KafkaBackendConsumer, once the application receives the first bulk of messages, it builds the plan according to it, and ignores the second read. The application should verify first that there are no more messages to read before deciding on the current state.

@yoniv87 yoniv87 added the bug Something isn't working label Nov 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
1 participant