You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
Is there somewhere where you make note of the kafka offsets that has been processed. Because I just sent 5 events to Kafka but it looks like the connector keeps on writing the 5 messages again and again in S3. Even though querying the iceberg table returns only the 5 records, the s3 objects keep increasing even though there are no new messages. Am I missing some parameter? Because 403 objects for just 5 events is a bit too much and if the connector is running, then the objects keep increasing even if there are no new kafka events
The text was updated successfully, but these errors were encountered:
haripriyarhp
changed the title
Re-writing the same data for each run even though there is no new data
Re-writing the same data for each run even though there is no new event in kafka
May 5, 2023
Hi,
Is there somewhere where you make note of the kafka offsets that has been processed. Because I just sent 5 events to Kafka but it looks like the connector keeps on writing the 5 messages again and again in S3. Even though querying the iceberg table returns only the 5 records, the s3 objects keep increasing even though there are no new messages. Am I missing some parameter? Because 403 objects for just 5 events is a bit too much and if the connector is running, then the objects keep increasing even if there are no new kafka events
The text was updated successfully, but these errors were encountered: