Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: added schema evolution to the merge statement #3135

Closed
wants to merge 158 commits into from
+596 −13
Closed

feat: added schema evolution to the merge statement #3135

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
158 commits
Select commit Hold shift + click to select a range
585dc57
messy pass of empty table update schema
Dec 13, 2024
edd4379
Enable readerFeatures in minReaderVersion 3
rjancewicz Nov 1, 2024
d690dfa
chore(deps): update which requirement from 6 to 7
dependabot[bot] Nov 4, 2024
65bc76d
perf: close partition writers concurrently to improve writes with man…
alexwilcoxson-rel Nov 8, 2024
e26591b
chore(deps): update thiserror requirement from 1 to 2
dependabot[bot] Nov 11, 2024
756af10
perf: batch json decode checkpoint actions when writing to parquet
alexwilcoxson-rel Nov 8, 2024
d1ab726
Update delta-lake-z-order.md
RyRyRyNguyen Nov 5, 2024
106564a
Update object_store to 0.10.2
thomasfrederikhoeck Nov 14, 2024
3042437
fix: cache credential resolution with the AWS credential provider
rtyler Nov 12, 2024
0db49c4
fix: jsonwriter should checkpoint by default
jusjosj Nov 13, 2024
0306275
valid table can have checkpoint as first log
stretchadito Nov 19, 2024
332027a
Fixed the deprecation warnings in spot check step of the build.
vksx Nov 19, 2024
596a571
chore: bump kernel
ion-elgreco Sep 14, 2024
0ead1a0
chore: adopt new datafusion crate
rtyler Sep 3, 2024
7d26a5d
fix: upgrade python ABI to 3.9 from 3.8
rtyler Sep 19, 2024
b20fecb
fix: adopt the right array item name which changed in kernel 0.3.1
rtyler Sep 20, 2024
0d97684
chore: drop python 3.8 support
ion-elgreco Sep 22, 2024
00f1210
chore: add a rust test demonstrating challenges with updates and lists
rtyler Oct 22, 2024
6b18e05
chore: upgrade to the latest datafusion 43
rtyler Nov 10, 2024
7187905
chore: upgrade all version ranges for the datafusion 43 ABI change
rtyler Nov 11, 2024
1d6dada
chore: add trait annotations for compilation with datafusion 43
rtyler Nov 11, 2024
fcca284
chore: bump kernel
ion-elgreco Sep 14, 2024
ed30755
fix: import DataType for the datafusion tests
rtyler Aug 19, 2024
74545f9
fix: address build failures with the newer kernel
rtyler Sep 19, 2024
131bc18
chore: datafusion 43 updates, use projected schema in some places
hntd187 Nov 15, 2024
99f1da8
fix: disable datafusion optimizers which perform list field name erasure
rtyler Nov 16, 2024
b278e9f
chore(docs): fix the verify table existence example in usage docs
vksx Nov 18, 2024
a8eb301
remove unnecessary backslash
vksx Nov 18, 2024
2e6e370
<fix>: decimal stat rounding overflow
gruuya Nov 4, 2024
1c779f1
feat: upgrade to delta_kernel 0.4.1 :tada:
rtyler Nov 21, 2024
b83cad4
chore: include license files in published crates
ankane Nov 21, 2024
635e39d
Update intro
Nov 19, 2024
796753d
Update intro
Nov 21, 2024
618613a
Add quickstart to intro page
Nov 21, 2024
4250de1
Add spaces
Nov 21, 2024
b869376
Add text
Nov 21, 2024
5973b8c
Add datafusion-ffi and order alphabetically
timsaucer Nov 21, 2024
a23ba7d
Add pycapsule interface to provide table provider to DataFusion
timsaucer Nov 21, 2024
a80034c
Add __datafusion_table_provider__ to pyi file
timsaucer Nov 22, 2024
24f6563
Fix date expr to str
Nov 22, 2024
55815a7
Fix python test
Nov 22, 2024
fa31baa
Fix python test
Nov 22, 2024
0cbf424
chore: fixed a bunch of warnings and deprecations
hntd187 Nov 22, 2024
befc60d
Fix num row statistics
lewiszlw Nov 13, 2024
a5579b4
docs: mention AWS_ENDPOINT_URL_DYNAMODB
maxitg Nov 23, 2024
1018c4d
feat: override dynamodb config
thomas-chauvet Nov 21, 2024
6fb1bff
fix(test): fix previous workaround
thomas-chauvet Nov 22, 2024
b7ae5ef
Fix JsonWriter and RecordBatchWriter to respect stats skipping
jjossick Nov 12, 2024
12393f3
chore: refactor JsonWriter to utilize the DeltaTable configuration more
rtyler Nov 22, 2024
e75011a
chore: bump the python version for release
rtyler Nov 23, 2024
32c8548
chore: refactor some of the writebuilder preconditions into the appro…
rtyler Nov 23, 2024
8c02180
fix: pass ending_timestamp correctly
ion-elgreco Nov 24, 2024
54c19c2
chore(deps): update hashbrown requirement from 0.14.3 to 0.15.2
dependabot[bot] Nov 25, 2024
9fe4a6a
fix: correct the schema definition of deletion vector
rtyler Nov 26, 2024
202362d
fix: workaround for Add actions being read slightly differently out o…
rtyler Nov 25, 2024
419a781
chore: bump key versions for the upcoming 0.22.1 release
rtyler Nov 28, 2024
449318d
fix: add nullability check in deltachecker
ion-elgreco Nov 24, 2024
9b99cf1
Implement query builder
PeterKeDer Nov 2, 2024
363ff6a
chore: add some more documentation to the new QueryBuilder interface
rtyler Nov 29, 2024
281f524
chore: bump for another python and core release
rtyler Nov 30, 2024
eb6da09
chore: missed a version update!
rtyler Nov 30, 2024
d293ff8
chore: add more robust regression tests for the checkpoint related fa…
rtyler Dec 3, 2024
9898993
fix: prevent attempting to read empty DVInfo
rtyler Dec 4, 2024
90cfeda
fix: add null checks when working with structs
ion-elgreco Dec 4, 2024
6df7867
chore: bump to 0.22.3 for another release
rtyler Dec 4, 2024
624890b
Added enable out of range to load_cdf
pblocz Dec 1, 2024
ab63ed4
Added enable out of range to python
pblocz Dec 1, 2024
e9d5711
Fixed fomatting and test in python plus renamed out of range flag for…
pblocz Dec 1, 2024
c5ddb20
Moved all boundary conditions before the main load_cdf loop
pblocz Dec 1, 2024
2f20bf8
add explicit type casts while merge
t1g0rz Dec 10, 2024
0812a29
fix No module named pandas
t1g0rz Dec 10, 2024
9d363c8
chore: bump delta-kernel to 0.5.0
roeap Dec 11, 2024
ad30383
chore: use uv
thomas-chauvet Dec 2, 2024
0165c6a
chore: lazy pyspark install
ion-elgreco Dec 15, 2024
dbcbf5a
fix: raise during checkpoint creating with without_files
ion-elgreco Dec 15, 2024
9dfee2b
chore: fmt
ion-elgreco Dec 15, 2024
713ee66
bulid out a plan for schema dift
Dec 16, 2024
e294044
change the projection to include either columns from source or target…
Dec 18, 2024
ca8b90e
bulid the schema based on the operations in the merge statement
Jan 8, 2025
7523514
add columns based on the operations statement and added merge flag
Jan 15, 2025
a12bde7
add columns based on the operations statement and added merge flag
Jan 15, 2025
a832247
feat: rust 2024 edition, moved unity to its own crate
hntd187 Nov 29, 2024
71af64c
feat: move unity catalog integration into its own crate
hntd187 Dec 15, 2024
b144bdf
feat: move unity catalog integration into its own crate
hntd187 Dec 15, 2024
2df39d3
feat: move unity catalog integration into its own crate
hntd187 Dec 15, 2024
6dc38c3
chore: bump versions for next release(s)
rtyler Dec 16, 2024
6498b90
chore: remove uv.lock which should not be checked in
rtyler Dec 16, 2024
5c70d85
fix: pin the 'home' crate version for 1.80 compatibility
rtyler Dec 17, 2024
0b5acca
chore: create a new minor version with the MSRV upgrade
rtyler Dec 19, 2024
2f2b4d7
chore: relocate peek_next_commit onto the logstore directly for easie…
rtyler Dec 17, 2024
5536379
fix: allow more than 15 concurrent transactions to have been committed
rtyler Dec 18, 2024
f54b047
fix: remove println from regular_rename func
tom1919 Dec 20, 2024
0eec173
fix: use projected_table_schema for projection in DeltaSchemaAdapter
jkylling Dec 18, 2024
c6c870e
fix: use correct aws_endpoint key
ion-elgreco Dec 20, 2024
f2267ab
fix: add test for Parquet column pruning in scans
jkylling Dec 22, 2024
35946e6
Add uv reference
FranArenas Dec 25, 2024
eff8cf7
fix: use of deprecated DeltaTable::peek_next_commit
jkylling Dec 22, 2024
fee6b82
chore: bump object store
ion-elgreco Dec 29, 2024
f13ffd8
fix: make version dynamic
ion-elgreco Dec 29, 2024
aa695df
refactor: make putifabsent default for s3
ion-elgreco Dec 29, 2024
e9b0f46
chore: add another test to cover the dynamdb logstore case
rtyler Dec 30, 2024
f3bfad3
fix: ensure defaulted options are set prior to ObjectStore creation
rtyler Dec 31, 2024
84baa0a
fix: add decimal expr parsing Signed-off-by: Ion Koutsouris
ion-elgreco Dec 29, 2024
f772ec0
chore: datafusion 44 fix for PartitionedFile
rtyler Dec 1, 2024
ac5bff8
chore: datafusion 44 prototype change on make_array
rtyler Dec 1, 2024
bccfa11
Pin to pre-release Datafusion
alamb Dec 21, 2024
fe8311c
chore: Update to latest sqlparser
alamb Dec 21, 2024
5e84754
chore: Reduce compiler warnings by updating to use non deprecated Dat…
alamb Dec 22, 2024
f169315
chore: bump kernel, datafusion 44rc1
ion-elgreco Jan 1, 2025
ffcd100
fix: add view scalar extraction
ion-elgreco Jan 2, 2025
04f3f5d
fix: introduce a reproduction case for List casting with polars
rtyler Dec 30, 2024
603cfa2
add except cols
Jan 4, 2025
c1f0a05
Added sql() and show() convenience method to QueryBuilder
DevChrisCross Jan 3, 2025
cee357a
feat: allow multiple Python threads to work with a single DeltaTable …
rtyler Jan 3, 2025
3cf3cbc
Rust API documentation
Abdullahsab3 Dec 29, 2024
22cfb93
chore(deps): update itertools requirement from 0.13 to 0.14
dependabot[bot] Jan 6, 2025
d98dbaa
fix: pass correct options to is_aws, reduce times sdk_config construc…
ion-elgreco Jan 8, 2025
e39606e
chore: bump the aws crate patch version for release
rtyler Jan 8, 2025
ff06eff
fix: reading cdf from a checkpointed table
hntd187 Jan 9, 2025
3212072
fix: reading cdf from a checkpointed table
hntd187 Jan 9, 2025
b93fb58
chore: raise patch versions for rust crate release
rtyler Jan 10, 2025
dc173eb
feat: object/log store operation_id aware
ion-elgreco Jan 10, 2025
3cc87ef
feat: asany logstore
ion-elgreco Jan 10, 2025
99de3d3
feat: custom execution handler
ion-elgreco Jan 10, 2025
ed4f13a
feat: ObjectStoreRegistry trait
ion-elgreco Jan 10, 2025
9a4e958
feat: deltalake-lakefs
ion-elgreco Jan 10, 2025
810641e
feat: lakefs integration python
ion-elgreco Jan 10, 2025
8d25d0a
chore: lakefs CI steps
ion-elgreco Jan 10, 2025
2f1a75d
chore: lakefs integration docs
ion-elgreco Jan 10, 2025
7a1fede
fix: don't rebuild each time uv is called
ion-elgreco Jan 10, 2025
5a58031
chore: disable lakefs for pyarrow engine
ion-elgreco Jan 10, 2025
ff27f93
fix: is_aws test flakyness
ion-elgreco Jan 10, 2025
53cda3e
refactor: kernel table features
ion-elgreco Jan 12, 2025
fcc3c0d
chore: update readme with latest state of project
ion-elgreco Jan 12, 2025
0a3015e
fix: ignore crc files when checking if provided path correspond to a …
guillotjulien Jan 12, 2025
ee3d3f9
fix: refine crc regex to only catch valid files
guillotjulien Jan 12, 2025
cbc2e6a
fix: docs rendering
Abdullahsab3 Jan 12, 2025
dc577dd
chore: allow kernel::visotors to be public for Snapshot usage
rtyler Oct 29, 2024
87bf3fd
chore: 0.24 py release
ion-elgreco Jan 14, 2025
5068c87
added schema mode to the python api and added some unit testing
Jan 15, 2025
4109aa5
refactor code and added filter in modify_schema
Jan 16, 2025
8097942
format python code and refactor
Jan 16, 2025
749f453
change updates unit test
Jan 16, 2025
13087ab
feat: retryable config enablement through storage_options
ion-elgreco Jan 14, 2025
8601d71
chore: clean up
ion-elgreco Jan 14, 2025
c0ef96c
chore: add docs
ion-elgreco Jan 14, 2025
a6d0824
fix: fmt scalar view values
ion-elgreco Jan 14, 2025
8e414d9
feat: generated columns
ion-elgreco Jan 12, 2025
2fdbd89
feat: enable generated columns merge
ion-elgreco Jan 12, 2025
954ad01
fix: cast generated col exprs always, don'fetch nested metadata
ion-elgreco Jan 12, 2025
7260698
fix: disallow new generated columns
ion-elgreco Jan 13, 2025
91ef75c
chore: generated expression parsing improvement, support on first wri…
ion-elgreco Jan 13, 2025
b918e65
chore: tests
ion-elgreco Jan 13, 2025
89194b4
Minor documentation typo
Milias Jan 14, 2025
4a716f9
feat(python, rust): add statistics_enabled to ColumnProperties
maxitg Jan 13, 2025
c8ce71e
remove max_statistics_size from ColumnProperties
maxitg Jan 14, 2025
d72d538
chore: preparing the 0.24 release of the Rust crates`
rtyler Jan 15, 2025
49480f9
fix merge conflict
Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading