Kafka Connect: Add delta writer support #12070
Conversation
(cherry picked from commit 2d4e680f20283efcd1064f8da33ea099133b171c)
(cherry picked from commit 12d44660fddc49fa0d60fa914ab9e8d1c9cb0867)
(cherry picked from commit c7651903a4a9daa1d5de6c5ecf4ddaa2573c0552)
(cherry picked from commit f40bac5b2179f1862df5924cec3ffaa11159f64f)
(cherry picked from commit 5331c8d8ffbb92551c95a6ea72abbe03ae4c01ae)
(cherry picked from commit 39982cbec35d7aeb767e07683ebc72eb1e5bec6c)
Apply spotless and fix method name
|
@bryanck copied over the code as is. Im planning to refactor upsert mode (delta writer) code, planning to add few improvements to it, potentially changing existing behavior. |
| private static final String TABLES_CDC_FIELD_PROP = "iceberg.tables.cdcField"; | ||
| private static final String TABLES_UPSERT_MODE_ENABLED_PROP = "iceberg.tables.upsertModeEnabled"; |
There was a problem hiding this comment.
i believe naming should change to dash-case-naming ? example cdc-field?
| /** | ||
| * This is modified from {@link org.apache.iceberg.util.StructProjection} to support record types. | ||
| */ | ||
| public class RecordProjection implements Record { | ||
|
|
||
| /** | ||
| * Creates a projecting wrapper for {@link Record} rows. | ||
| * | ||
| * <p>This projection does not work with repeated types like lists and maps. |
There was a problem hiding this comment.
Should we add RecordProjection class to iceberg.data or iceberg.core for other downstream projects to use it?
| import org.apache.iceberg.relocated.com.google.common.collect.Sets; | ||
| import org.apache.iceberg.types.TypeUtil; | ||
|
|
||
| abstract class BaseDeltaTaskWriter extends BaseTaskWriter<Record> { |
There was a problem hiding this comment.
@bryanck should we add this and other DeltaWriter* classes to core, next to AppendWriter classes? for downstream projects to reuse it?
using RecordWrapper + Operation it becomes generic enough to be in core? what do you think?
example refactored version:
BaseDeltaTaskWriter
There are a couple of discussions on why we didn't originally add the delta writer functionality. I think we will need to resolve those discussions before we add this. |
resolves #10842