feat(python): Experimental credential provider support for Delta read/scan/write #20842
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds automatic Python-side credential provider initialization to the Delta I/O functions (
read_delta
,scan_delta
,write_delta
), similar to those previously added to other I/O functions. This includesDefaultAzureCredential
(ref #20808):Before calling into the Delta Python library, we directly call the credential provider on the Python-side and inject them into
storage_options
, as the Delta Python library does not accept a callable credential provider. This works as our built-in credential providers (AWS
,Azure
,GCP
) all returnobject_store
-compatible key-value pairs - i.e. it's valid to do something like this: