Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-writing the same data for each run even though there is no new event in kafka #33

Open
haripriyarhp opened this issue May 5, 2023 · 0 comments
Open

Re-writing the same data for each run even though there is no new event in kafka #33

haripriyarhp opened this issue May 5, 2023 · 0 comments

Comments

@haripriyarhp
Copy link

Hi,
Is there somewhere where you make note of the kafka offsets that has been processed. Because I just sent 5 events to Kafka but it looks like the connector keeps on writing the 5 messages again and again in S3. Even though querying the iceberg table returns only the 5 records, the s3 objects keep increasing even though there are no new messages. Am I missing some parameter? Because 403 objects for just 5 events is a bit too much and if the connector is running, then the objects keep increasing even if there are no new kafka events

image
@haripriyarhp haripriyarhp changed the title Re-writing the same data for each run even though there is no new data Re-writing the same data for each run even though there is no new event in kafka May 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@haripriyarhp