Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support metadata tables #823

Open
xxchan opened this issue Dec 18, 2024 · 5 comments
Open

feat: support metadata tables #823

xxchan opened this issue Dec 18, 2024 · 5 comments
Labels
enhancement New feature or request

Comments

@xxchan
Copy link
Contributor

xxchan commented Dec 18, 2024

In Iceberg Spark/Flink there are metadata tables that provide information around the table: https://iceberg.apache.org/docs/latest/spark-queries/#inspecting-tables

Supporting this in iceberg-rust allows other engines (like RisingWave) to support these "standard" metadata tables.

reference implementation:
- https://py.iceberg.apache.org/api/#inspecting-tables
- https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/inspect.py#L58-L90

List of all metadata tables: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/MetadataTableType.java#L23

@Fokko
Copy link
Contributor

Fokko commented Dec 18, 2024

Thanks for working on this @xxchan. Metadata tables are very important when it comes to the metadata operations. For example, the snapshots table in #822 will make it easy to perform operations like expiring snapshots.

@flaneur2020
Copy link
Contributor

i'd like to contribute to some of the implementation of these metadata tables, similar to the work done by @rshkv based on @xxchan's branch.

I plan to start with the MANIFESTS table first. could you kindly let me know if anyone else is currently working on this or has plans to do so? I want to avoid any potential conflicts. thanks.

@rshkv
Copy link
Contributor

rshkv commented Dec 28, 2024

@flaneur2020, apologies, I was actually just working on manifests and I should be able to put that up in a few days. If you want to go ahead with entries or files there'll be some overlap and code to reuse but that's fine - happy to rebase.

Ignore the above. I'm looking at entries now. Not working on manifests.

@DeaconDesperado
Copy link
Contributor

I'd like to attempt the FILES metadata table if not already being worked on. Thanks!

@liurenjie1024
Copy link
Contributor

I'd like to attempt the FILES metadata table if not already being worked on. Thanks!

Welcome to contribute!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

6 participants