2024-10-03 kernel meeting notes

[zachschuermann]

summary

0.4 release upcoming need to plan/sync with delta-rs.

pull requests/ongoing threads

• removing default features! change default features to be none #339
• typed table PR
• lots more prs...

attendees

@nicklan @zachschuermann @hntd187 @OussamaSaoudi-db

notes stream

To discuss: (1) integration testing to unblock release and get delta-rs the changes they need (2) design review from @hntd187

• push (1) to next week/offline. nick has a PR up for integration testing. also planning to leverage cargo-semver-checks.
• (2) nullability design from @hntd187
  • https://github.com/delta-incubator/delta-kernel-rs/blob/beeb6e800c259d47ade2e9814f02ddeb89df68c1/kernel/src/engine/parquet_stats_skipping.rs#L28-L71
  • extend binary operator to have default = (not null safe) and a null-safe =?
  • reasonable for engine? but if we require it then this demands all engines support it.
  • note spark under the hood: replace a = b -> a is not null and b is not null and a = b (spark only emits null checks for nullable columns)
  • we could do this in kernel
  • for ryan in data skipping: if engine is doing null-safe of where clause then kernel needs to have diff semantics in data skipping
  • note this is opt-in: engine has to choose to evaluate expression as null-safe instead of normal
  • WHERE clause is WHERE <expr>: what actually gets evaluated: WHERE <expr> <=> true (defined so that nulls are comparable)
  • data skipping push <=> true down to ands/ors
  • steve: do we want to provide a way for engines to do this where they see fit? we need this facility (ability to do nullsafe comparisons). there's a way to express this just by doing rewrites, but problem is you replicate the whole expression. many engines will already have specific <=> expression. if we do the rewrite we are pessimizing performance.
  • ryan: for skipping we have to push it down to get accurate skipping.
  • main ask: how to move forward right now? could take on optimal approach later and just do rewrites? what is the most useful library API for engines? not clear. flexibility is good but don't want to be overly flexible
  • we handoff expr to engine, then they run visitors in order to translate to their own expression. if we add a visit_nullsafe_eq: duckdb can take this and turn into IS NOT NULL and <expr> = true (if they dont have <=>) or if they do, then they just emit <=>. What about rust engines? if they look directly at the expression enum? maybe introduce an API like 'hey can you translate this'. note that if transform is trivial, probably could just document it.
  • nick strawman: do not add nullsafe operator initially, do rewrite internally (no change from today). When we want to generate expression for engine to use. is there a case where the engine needs to know it needs to do nullsafe.
  • ryan: fuzzy on this - parquet footer skipping PR: two different entrypoints (apply_expr and apply sql where - you opt in to whichever one you want). second PR assumes everyone wants apply sql where (default engine/parquet arrow - we say our default engine has sql where clause)
  • nick: punt on whether we have to add this other expression? can see how far this takes us. engines might have to do a little bit of work
  • avoid nullsafe eq: we don't have nullsafe other operators. it's a special case in spark. do we want to take on a special case here? initial reaction: no.

bit from zach: quick check on commit_info API :) and what we want to call Transaction

# commit info
{ "commitInfo": {"engine_one": "data"}, {"kernel_version": "v0.3.1"} }

spec allows almost anything but inCommitTimestamp requires structure. In practice, there is a defined schema.