refactor: Make CDF use TableConfiguration and refactor log replay #645

OussamaSaoudi · 2025-01-15T00:09:29Z

What changes are proposed in this pull request?

This PR replaces the old CDF protocol and metadata checks to use TableConfiguration. This paves the way for checking deletion vector enablement, in-commit timestamp enablement, and verifying column mapping support in the future.

This PR also refactors the LogReplay logic to simplify its design. The LogReplayScanner is removed. Instead, each commit file is mapped to a ProcessedCdfCommit, which can then be mapped to Iterator<TableChangesScanBatch>

Depends on: #644
Please only look at these commits

How was this change tested?

This is a refactor. All existing tests pass.

codecov · 2025-01-15T00:13:35Z

Codecov Report

Attention: Patch coverage is 91.08527% with 23 lines in your changes missing coverage. Please review.

Project coverage is 84.31%. Comparing base (3305d3a) to head (7243c36).

Files with missing lines	Patch %	Lines
kernel/src/table_changes/log_replay.rs	89.51%	0 Missing and 15 partials ⚠️
kernel/src/actions/visitors.rs	41.66%	0 Missing and 7 partials ⚠️
kernel/src/table_changes/scan.rs	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #645      +/-   ##
==========================================
+ Coverage   84.22%   84.31%   +0.09%     
==========================================
  Files          77       77              
  Lines       17694    17772      +78     
  Branches    17694    17772      +78     
==========================================
+ Hits        14902    14984      +82     
+ Misses       2080     2075       -5     
- Partials      712      713       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

scovich · 2025-01-15T16:05:28Z

kernel/src/table_changes/log_replay/tests.rs

+        configuration: HashMap::from([
+            ("delta.enableChangeDataFeed".to_string(), "true".to_string()),
+            (
+                "delta.enableDeletionVectors".to_string(),
+                "true".to_string(),
+            ),
+            ("delta.columnMapping.mode".to_string(), "none".to_string()),


aside: I wish rust had better (more transparent) handling of String vs. &str... code like this gets so ugly and unwieldy.

yep the to_string everywhere is kind of gross, but at least we're aware of where the allocations are happening

scovich · 2025-01-15T16:06:37Z

kernel/src/table_changes/log_replay/tests.rs

@@ -28,6 +28,30 @@ fn get_schema() -> StructType {
    ])
 }

+fn table_config() -> TableConfiguration {
+    let schema_string = serde_json::to_string(&get_schema()).unwrap();
+    let metadata = Metadata {


Out of curiosity, why is Metadata infallible while Protocol is fallible? Seems like any number of things could go wrong with it, such as partition column names that aren't actually in the schema?

Part of me wonders if we should make a clean break between P&M as "plain old data" vs. table configuration as the one stop shop for validation of that raw data? So then, any time you see a P or M in isolation, you have to assume it's broken in arbitrary ways. It's only "for sure" self-consistent and valid if a TC successfully wrapped it.

I think Metadata just doesn't have a constructor for kernel/rust data that does these checks. All it has is pub fn try_new_from_data(data: &dyn EngineData)

Part of me wonders if we should make a clean break between P&M as "plain old data" vs. table configuration as the one stop shop for validation of that raw data?

Ohhh so proposing that we move the validation that we do in Protocol::try_new to TC? Then this Metadata validation could also live there.

Moreover if we think of protocol as just "raw, unchecked data", I'm starting to wonder if we should move ensure_read_supported, has_writer_feature and has_reader_feature to TC?

Yeah... TC becomes the logical heart of the table, with P&M as simple raw information underneath

Note for anyone reading this thread, I'm tracking table config stuff in #650

scovich · 2025-01-15T16:10:59Z

kernel/src/table_changes/mod.rs

            schema,
+            table_configuration: start_snapshot.table_configuration().clone(),


How does schema differ from table_configuration.schema()?
I'm seeing this pattern a lot in this new table configuration code. If they're the same, we should remove the redundant arg; otherwise we should document clearly why/how they could differ.

Schema is the schema of the CDF (taken from the end version). This table_configuration is from the start version. I'll change the naming to start_table_config to be clearer.

scovich · 2025-01-15T16:13:53Z

kernel/src/table_changes/scan_file.rs

+            Some([ReaderFeatures::DeletionVectors]),
+            Some([ReaderFeatures::ColumnMapping]),


Should one of those be WriterFeatures? Also, the spec says:

Reader features should be listed in both readerFeatures and writerFeatures simultaneously, while writer features should be listed only in writerFeatures. It is not allowed to list a feature only in readerFeatures but not in writerFeatures.

This seems to be the case for all reader features. If that's the case, then ensure_read_supported should be checking that the reader features exist in both the reader and and the writer features list for a given protocol.

scovich · 2025-01-15T16:14:51Z

kernel/src/table_configuration.rs

@@ -39,34 +40,32 @@ impl TableConfiguration {
            column_mapping_mode,
        })
    }
-    pub fn with_protocol(self, protocol: impl Into<Option<Protocol>>) -> DeltaResult<Self> {
+    pub fn with_protocol(


aside: this was in the parent diff as well. Either the link in the PR description has the wrong set of commits, or this code leaked accidentally into the previous PR?

scovich · 2025-01-15T16:17:11Z

kernel/src/table_changes/log_replay.rs

+    // Note: We do not perform data skipping yet because we need to visit all add and
+    // remove actions for deletion vector resolution to be correct.
+    //
+    // Consider a scenario with a pair of add/remove actions with the same path. The add


NOTE: Indentation change, best reviewed with whitespace changes hidden:

kernel/src/table_changes/log_replay.rs

OussamaSaoudi · 2025-01-17T17:10:12Z

kernel/src/table_changes/scan_file.rs

+            Some::<Vec<String>>(vec![]),
+            Some::<Vec<String>>(vec![]),


Note: Creating Protocol from None is actually pretty challenging since it doesn't know the type of T in Option::<T>::None. Also, the type try_new takes for reader/writer features is Option<impl IntoIterator<Item = impl Into<String>>> shouldn't we be using impl ToString?

Into<String> and ToString have different characteristics, especially wrt String:

Into<String> for String is a move

ToString for String makes a copy

More types impl ToString than impl Into<String>

Dunno which we "should" use here...

Hmm i guess we should prefer moves. Though this is affected by the proposed changes to move all logic to TableConfiguration, so I'll leave this for now and address it in #650

OussamaSaoudi · 2025-01-19T01:28:24Z

kernel/src/table_changes/log_replay.rs

-            let string_list: DataType = ArrayType::new(STRING, false).into();
-            let string_string_map = MapType::new(STRING, STRING, false).into();
+            let str_list: DataType = ArrayType::new(STRING, false).into();
+            let str_str_map: DataType = MapType::new(STRING, STRING, false).into();


Note: Renaming so that the lines below don't get too long.

scovich

LGTM, except one possible latent bug to double-check

scovich · 2025-01-21T22:30:33Z

kernel/src/table_changes/log_replay.rs

+        visitor.visit_rows_of(actions.as_ref())?;
+        let has_metadata = visitor.metadata.is_some();
+        match (visitor.protocol, visitor.metadata) {
+            (None, None) => {}


Suggested change

(None, None) => {}

(None, None) => {} // no change

scovich · 2025-01-21T22:31:16Z

kernel/src/table_changes/log_replay.rs

+        match (visitor.protocol, visitor.metadata) {
+            (None, None) => {}
+            (p, m) => {
+                let p = p.unwrap_or_else(|| table_configuration.protocol().clone());


Suggested change

let p = p.unwrap_or_else(|| table_configuration.protocol().clone());

// at least one of P&M changed, so update the table configuration

let p = p.unwrap_or_else(|| table_configuration.protocol().clone());

scovich · 2025-01-21T22:32:07Z

kernel/src/table_changes/log_replay.rs

+                    m,
+                    p,


Given that we're anyway putting these on their own lines, probably don't need the let any more. Just move the code here directly.

scovich · 2025-01-21T22:33:03Z

kernel/src/table_changes/log_replay.rs

-            // The only (path, deletion_vector) pairs we must track are ones whose path is the
-            // same as an `add` action.
-            remove_dvs.retain(|rm_path, _| add_paths.contains(rm_path));
+        if has_metadata {


Maybe this makes it more self-describing?

Suggested change

if has_metadata {

if metadata_changed {

Would this be better as this?

require!( !metadata_changed || table_schema.as_ref() == table_configuration.schema(), Error::change_data_feed_incompatible_schema( table_schema, table_configuration.schema() ) );

Hmm this may encourage an expensive swap

table_schema.as_ref() == table_configuration.schema() || !metadata_changed

I'll keep it using the if for now.

scovich · 2025-01-21T22:35:26Z

kernel/src/table_changes/log_replay.rs

+                getters[14].get_int(i, "protocol.min_reader_version")?
            {
                let protocol =
-                    ProtocolVisitor::visit_protocol(i, min_reader_version, &getters[12..=15])?;
+                    ProtocolVisitor::visit_protocol(i, min_reader_version, &getters[14..=17])?;


This looks wrong (existing bug)? Wouldn't we read [14] twice (or [12] previously)?
Or does it demand the full list even tho it won't use the first one?

visit_protocol expects all fields of protocol to be passed in, even though it only reads elements 1,2, 3 (ignores 0).

visit_protocol:

require!( getters.len() == 4, Error::InternalError(format!( "Wrong number of ProtocolVisitor getters: {}", getters.len() )) ); let min_writer_version: i32 = getters[1].get(row_index, "protocol.min_writer_version")?; let reader_features: Option<Vec<_>> = getters[2].get_opt(row_index, "protocol.reader_features")?; let writer_features: Option<Vec<_>> = getters[3].get_opt(row_index, "protocol.writer_features")?; Protocol::try_new( min_reader_version, min_writer_version, reader_features, writer_features, )

I could change this, but maybe that could be another PR?

No need -- I was just verifying that we didn't have a double-use of that field by mentioning it twice.

github-actions bot assigned OussamaSaoudi Jan 15, 2025

OussamaSaoudi force-pushed the table_changes_refactor branch from 774fd50 to 23f4b6f Compare January 15, 2025 00:20

scovich reviewed Jan 15, 2025

View reviewed changes

OussamaSaoudi force-pushed the table_changes_refactor branch from 95cac9c to 5474769 Compare January 17, 2025 06:08

github-actions bot added the breaking-change Change that will require a version bump label Jan 17, 2025

OussamaSaoudi commented Jan 17, 2025

View reviewed changes

OussamaSaoudi force-pushed the table_changes_refactor branch from 5474769 to 436852b Compare January 19, 2025 01:17

OussamaSaoudi commented Jan 19, 2025

View reviewed changes

OussamaSaoudi marked this pull request as ready for review January 19, 2025 01:48

OussamaSaoudi requested review from zachschuermann, nicklan and scovich January 19, 2025 01:48

OussamaSaoudi changed the title ~~[wip] refactor: Make CDF use TableConfiguration and refactor log replay~~ refactor: Make CDF use TableConfiguration and refactor log replay Jan 19, 2025

OussamaSaoudi removed the breaking-change Change that will require a version bump label Jan 19, 2025

scovich approved these changes Jan 21, 2025

View reviewed changes

OussamaSaoudi force-pushed the table_changes_refactor branch from fae5a21 to 0ece561 Compare January 22, 2025 01:03

github-actions bot added the breaking-change Change that will require a version bump label Jan 22, 2025

OussamaSaoudi added 12 commits January 27, 2025 19:22

table-changes refactor first draft

06aaca7

fix tests for refactor

859d25a

Read fewer metadata fields

7cced5f

Stop constructing new schema each time

a659bac

remove import warnings

9f91018

refactor to reduce LOC diff

4f2e246

Rename processed CDF commit

51ce1e5

Move table_config check

3888c0d

addressed more pr comments

88c6fdf

Shortened code

561f454

Update to new table_configuration

2f5e510

Addressing nits

c1b3b98

address PR comments

eb925a1

OussamaSaoudi force-pushed the table_changes_refactor branch from 0ece561 to eb925a1 Compare January 28, 2025 03:23

fix naming

7243c36

OussamaSaoudi force-pushed the table_changes_refactor branch from 552c5f0 to 7243c36 Compare January 29, 2025 02:03

OussamaSaoudi mentioned this pull request Jan 29, 2025

[WIP] Move protocol validation to TableConfiguration #665

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Make CDF use TableConfiguration and refactor log replay #645

refactor: Make CDF use TableConfiguration and refactor log replay #645

OussamaSaoudi commented Jan 15, 2025 •

edited

Loading

codecov bot commented Jan 15, 2025 •

edited

Loading

scovich Jan 15, 2025

OussamaSaoudi Jan 17, 2025

scovich Jan 15, 2025

scovich Jan 15, 2025

OussamaSaoudi Jan 17, 2025

OussamaSaoudi Jan 17, 2025

scovich Jan 17, 2025

OussamaSaoudi Jan 28, 2025

scovich Jan 15, 2025

OussamaSaoudi Jan 17, 2025

scovich Jan 15, 2025

OussamaSaoudi Jan 17, 2025

scovich Jan 15, 2025

scovich Jan 15, 2025

OussamaSaoudi Jan 17, 2025

scovich Jan 17, 2025

OussamaSaoudi Jan 19, 2025

OussamaSaoudi Jan 19, 2025

scovich left a comment

scovich Jan 21, 2025

scovich Jan 21, 2025

scovich Jan 21, 2025

scovich Jan 21, 2025

OussamaSaoudi Jan 22, 2025

OussamaSaoudi Jan 22, 2025

scovich Jan 21, 2025

OussamaSaoudi Jan 22, 2025

scovich Jan 22, 2025 •

edited

Loading

		schema,
		table_configuration: start_snapshot.table_configuration().clone(),

		Some([ReaderFeatures::DeletionVectors]),
		Some([ReaderFeatures::ColumnMapping]),

	let p = p.unwrap_or_else(\|\| table_configuration.protocol().clone());
	// at least one of P&M changed, so update the table configuration
	let p = p.unwrap_or_else(\|\| table_configuration.protocol().clone());

refactor: Make CDF use TableConfiguration and refactor log replay #645

Are you sure you want to change the base?

refactor: Make CDF use TableConfiguration and refactor log replay #645

Conversation

OussamaSaoudi commented Jan 15, 2025 • edited Loading

What changes are proposed in this pull request?

How was this change tested?

codecov bot commented Jan 15, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scovich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

scovich Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

OussamaSaoudi commented Jan 15, 2025 •

edited

Loading

codecov bot commented Jan 15, 2025 •

edited

Loading

scovich Jan 22, 2025 •

edited

Loading