Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python): Experimental credential provider support for Delta read/scan/write #20842

Merged
merged 1 commit into from
Jan 23, 2025

Conversation

nameexhaustion
Copy link
Collaborator

@nameexhaustion nameexhaustion commented Jan 22, 2025

Adds automatic Python-side credential provider initialization to the Delta I/O functions (read_delta, scan_delta, write_delta), similar to those previously added to other I/O functions. This includes DefaultAzureCredential (ref #20808):

Before calling into the Delta Python library, we directly call the credential provider on the Python-side and inject them into storage_options, as the Delta Python library does not accept a callable credential provider. This works as our built-in credential providers (AWS, Azure, GCP) all return object_store-compatible key-value pairs - i.e. it's valid to do something like this:

creds, _expiry = pl.CredentialProviderAzure()()

pl.read_delta(
    "abfss://...",
    storage_options=creds,
)

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars labels Jan 22, 2025
@ion-elgreco
Copy link
Contributor

Just fyi, inside delta-rs we have this for AWS actually

Copy link

codecov bot commented Jan 23, 2025

Codecov Report

Attention: Patch coverage is 40.00000% with 24 lines in your changes missing coverage. Please review.

Project coverage is 79.64%. Comparing base (b333ab7) to head (240aab3).
Report is 43 commits behind head on main.

Files with missing lines Patch % Lines
py-polars/polars/io/cloud/credential_provider.py 25.00% 12 Missing ⚠️
py-polars/polars/dataframe/frame.py 36.36% 5 Missing and 2 partials ⚠️
py-polars/polars/io/delta.py 58.33% 3 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #20842      +/-   ##
==========================================
- Coverage   79.64%   79.64%   -0.01%     
==========================================
  Files        1571     1571              
  Lines      223165   223200      +35     
  Branches     2546     2554       +8     
==========================================
+ Hits       177737   177757      +20     
- Misses      44844    44855      +11     
- Partials      584      588       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@nameexhaustion
Copy link
Collaborator Author

Just fyi, inside delta-rs we have this for AWS actually

Nice! We unfortunately aren't able to use it at the moment as we don't have rust-level integration with delta.

@nameexhaustion nameexhaustion force-pushed the credential-provider-delta branch from 1bae22b to 240aab3 Compare January 23, 2025 09:43
@nameexhaustion nameexhaustion marked this pull request as ready for review January 23, 2025 10:20
@ritchie46 ritchie46 merged commit 6eafeb3 into pola-rs:main Jan 23, 2025
14 of 15 checks passed
@c-peters c-peters added the accepted Ready for implementation label Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants