2025-01-09 kernel meeting notes

[OussamaSaoudi-db]

summary

pull requests/ongoing threads

• @zachschuermann on write FFI, user guides, parquet reader improvement (async), need a way to create engine data
• @OussamaSaoudi-db Introducing in-commit timestamp support
• @hntd187 has Map PR
• @roeap delta-rs integration! Finding a transition path to bring it in incrementally.
• @roeap prototyping a datafusion x delta-kernel-rs integration. PR incoming.
• @roeap /in list is needed by delta-rs. Looking at how to add that
• @scovich Helping out with data skipping and partition pruning. Designing the write path.
• @scovich + @nicklan adding catalog support in kernel. Accessing tables through the catalog instead of the filesystem.
• @nicklan Exposing expression for physical to logical transformation to engines.
• @timsaucer Expanding data skipping to use partition columns
• @zachschuermann Feature flag default-engine-rustls for rust-defined TLS.
• @zachschuermann 0.6.1 release incoming!
• @zachschuermann FFI work to begin expanding write path to engines/consumers outside of rust. [wip][ffi] transaction and append data FFI #550

attendees

@nicklan @hntd187 @zachschuermann @OussamaSaoudi-db @roeap @dennyglee @scovich @maruschin @mightyshazam

leftovers

• zach: define MSRV policy (follow datafusion?)
• zach: define release expectation
• zach: labels discussion
• zach/robert: datafusion integration
• zach/oussama: Address the reqwest dependency that uses native TLS.

notes stream

• robert: looking to get feedback on the datafusion x delta-kernel-rs integration. They already pass dat tests! 🎉
• nick: Take a look at the data transform PRs (parts 1-4). Hopefully one of the last major changes to the read path. Possible 1.0
• Zach: Introducing feature flag for default engine that changes dependencies we pull in for reqwest. Uses rust-tls instead of native tls. This was a delta-rs ask for AWS lambda. PTAL at the PR. 0.6.1 release depends on this.
• nick: We need to simplify feature flag story since it's starting to get tricky.
• nick: Current state of kernel lets you either do all of the read in the kernel, or makes the engine do the work of figuring out expression transforms. the transform work communicates physical to logical transforms to the engine so that engine can participate more
• kyjah: Dot net requires more write support from the kernel.
• zach: Status on write/write ffi: currently implement blind append support in the kernel. We don't have the ability to create tables just yet. This exist in the kernel and rust-side apis. FFI apis are in a PR. The flow is create transaction -> write parquet -> hand metadata to the transaction -> ask kernel to commit the transaction. POC Pr gives a good start. Productionized version incoming.
• stephen: Should expressions over map types by reworked entirely? Seems that there's a lot of changes in flight.
• robert: Stephen brought up credential vendoring in delta-rs, which will be good to look at. Currently making assumptions that all data resides in the same storage account. Datafusion avoids making this assumption because data/metadata could be spread across accounts.
• Stephen: blind appends write relative paths instead of full paths. This breaks delta-rs's ability to read the paths later on. This was a delta-rs bug. Kernel should be able to write files without having to worry about paths, so this should be fixed on the delta-rs side.
• Ryan: 2 reactions. 1) on map support. It seems like we'll be able to do partition pruning without map support, we may leave it aside for now until we need it again. We might need map to point to engine data instead of data in kernel's memory. This allows unbounded kernel memory, which we want to avoid. 2) Engine has to handle paths correctly. shallow clones and credential vending must be correct. Engine should not pass relative paths to kernel, nor should we relativize them. Delta sharing passes urls as a delta table, and those are absolute paths, not relative.
• Robert: Current impl with presigned urls just does http and doesn't care about relative paths.
• Kyjah: question on datafusion + kernel. Is the goal to hit parity with delta-rs.
• zach: delta-rs is a consumer of datafusion. delta-rs needs to do some computation and pass commands to datafusion. The datafusion integration in these notes refers to integrating kernel into datafusion. Making it into a TableProvider so that datafusion can directly read/write delta tables. delta-rs has implemented some of these traits. Question: What apis do consumers want to use. Can interface with datafusion to access delta tables, or go through delta-rs which has a specific api for delta and will let you specify an engine.
• Kyjah: Goal is to let users utilize the api they want, allowing them to pass in query plan + record batch. Looking for guidance on how to use kernel.
• Robert: delta-rs's operations bulider lets you construct delta operations and converts them into query plans. The part that actually reads the delta log should eventually be kernel. The goal is to push delta where operations back, and instead focus on the operations api which kernel doesn't provide.
• zach: Kyjah is running into the high flexibility, but low opinions on which engine/api to use. Consider 2 decisions: 1) what apis do you want to interact with. Query plans, or some higher level api? 2) What engine do you want to leverage for that? Ex: Rust api + datafusion could be satisfied with kernel apis, or delta-rs apis, then pulling datafusion as an engine.
• Kyjah: kernel is the correct way to interact with the delta table. Question is what will be implemented and what will be open for extension. For now the questions are answered, but obviously things are in flux.
• Robert: Will kernel expose something that is more tailored to exposing the log details like files for vacuum. Usecas is to perform optimizations. Need to peer into table to see what the current state is. For delta-rs, this featureset would be very useful to be able to probe the data. This work would be an easier way to integrate kernel/delta-rs. Without it, it's hard to hit parity between the two. Scan data api is not sufficient.
• Nick: Take a look at inspect-table. This prints out every action.
• Robert: This is exactly what we need.
• nick: You need to build your own visitor, but you are able to visit everything. We may want to move some of this into the kernel. We do want to expose this, but not in the default set of flags. doing so is subject to breakage if the protocol updates.
• robert: we already use visitor pattern, so this may be convenient than you'd think
• Robert: QOL of developing for kernel has significantly improved over time :D