Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support GCP Auth #444

Merged
merged 3 commits into from
Nov 22, 2023
Merged

Support GCP Auth #444

merged 3 commits into from
Nov 22, 2023

Conversation

edwardfeng-db
Copy link
Contributor

Changes

  • Support auth in GCP
    • With Google Credentials
    • With Google Service Account

Tests

  • make test run locally
  • make fmt applied
  • relevant integration tests applied

@codecov-commenter
Copy link

codecov-commenter commented Nov 14, 2023 •
edited
Loading

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (2b952c9) 51.67% compared to head (38cf1b7) 51.62%.
Report is 1 commits behind head on main.

Files Patch % Lines
databricks/sdk/core.py 23.80% 32 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #444      +/-   ##
==========================================
- Coverage   51.67%   51.62%   -0.06%     
==========================================
  Files          38       38              
  Lines       22277    22318      +41     
==========================================
+ Hits        11512    11521       +9     
- Misses      10765    10797      +32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mgyucht
mgyucht reviewed Nov 21, 2023
View reviewed changes
Copy link
Contributor

@mgyucht mgyucht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought I left the review but haven't apparently. Some minor notes, but this is very close.

databricks/sdk/__init__.py Outdated Show resolved Hide resolved
databricks/sdk/core.py Outdated Show resolved Hide resolved
setup.py Outdated Show resolved Hide resolved
databricks/sdk/core.py
@@ -265,6 +273,70 @@ def refreshed_headers() -> Dict[str, str]:
return refreshed_headers


@credentials_provider('google-credentials', ['host', 'google_credentials'])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this is the same as with the Go library. But in theory, google_credentials does not even need to be specified, because there is also a default directory that the google-auth library looks in, if I'm not mistaken. We might be able to remove this and allow users to auto-login with their google credentials set up via the default app credentials pathway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I thought so as well at the beginning but after checking more I think we do need to point it to the service account json file, which is different from the default auth json file as the service account one contains more info about the keys and secrets for signing the jwt etc. So it seems like we need this to make it work

mgyucht
mgyucht approved these changes Nov 21, 2023
View reviewed changes
Copy link
Contributor

@mgyucht mgyucht left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM provided nightly tests pass!

@edwardfeng-db edwardfeng-db added this pull request to the merge queue Nov 22, 2023
Merged via the queue into main with commit d8da7a8 Nov 22, 2023
9 checks passed
@edwardfeng-db edwardfeng-db deleted the edwardfeng-db/gcp-auth branch November 22, 2023 12:39
@edwardfeng-db edwardfeng-db mentioned this pull request Nov 22, 2023
Closed
3 tasks
@mgyucht mgyucht mentioned this pull request Nov 29, 2023
Merged
mgyucht added a commit that referenced this pull request Nov 29, 2023
@mgyucht
Release v0.14.0
5762f97 
Major changes:
* GCP Auth is now supported in the Python SDK. To use Google credentials-based authentication, specify your Default Application Credentials in the `GOOGLE_CREDENTIALS` environment variable or corresponding `google_credentials` parameter in `Config` or the client constructors. You may provide either the path to the file containing your credentials or the credentials themselves serialized as JSON. To use Google impersonation, specify the service principal to impersonate in the `DATABRICKS_GOOGLE_SERVICE_ACCOUNT` environment variable or the corresponding `google_service_account` parameter in `Config` or the client constructors. See [#444](#444) for the changes.

Bug fixes:
* Fix flask app example ([#445](#445)).
* Fix deserialization of repeated enums ([#450](#450), [#452](#452)).
* Capture stdout and stderr separately when calling Azure CLI ([#460](#460)).

Other changes:
* Change the name of retries logger to `databricks.sdk.retries` ([#453](#453)).

API Changes:

 * Added `pipeline_id` field for `databricks.sdk.service.catalog.TableInfo`.
 * Added `enable_predictive_optimization` field for `databricks.sdk.service.catalog.UpdateCatalog` and `databricks.sdk.service.catalog.UpdateSchema`.
 * Removed `databricks.sdk.service.catalog.UpdatePredictiveOptimization` and `databricks.sdk.service.catalog.UpdatePredictiveOptimizationResponse` dataclasses.
 * Removed `enable_optimization()` method for [w.metastores](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/metastores.html) workspace-level service.
 * Added `description` field for `databricks.sdk.service.jobs.CreateJob`  and `databricks.sdk.service.jobs.JobSettings`.
 * Added `list_network_connectivity_configurations()` and `list_private_endpoint_rules()` methods for [a.network_connectivity](https://databricks-sdk-py.readthedocs.io/en/latest/account/network_connectivity.html) account-level service.
 * Added `databricks.sdk.service.settings.ListNccAzurePrivateEndpointRulesResponse`, `databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsRequest`, `databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsResponse`, and `databricks.sdk.service.settings.ListPrivateEndpointRulesRequest` dataclasses.

Internal changes:

* Make ucws tests skipped when DATABRICKS_ACCOUNT_ID is present ([#448](#448)).

OpenAPI SHA: 22f09783eb8a84d52026f856be3b2068f9498db3, Date: 2023-11-23
Dependency updates:

 * Bump API spec for Python SDK ([#454](#454)).
pietern
pietern reviewed Nov 29, 2023
View reviewed changes
README.md
google_service_account=input('Google Service Account: '))

```

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edwardfeng-db Could you take a second pass at this doc? The field names mentioned are the Go SDK field names, not the Python SDK field names (PascalCase vs snake_case).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I see, sorry about this, let me update that. Some copy pasta issues

github-merge-queue bot pushed a commit that referenced this pull request Nov 29, 2023
@mgyucht
Release v0.14.0 (#463)
8da97de 
Major changes:
* GCP Auth is now supported in the Python SDK. To use Google
credentials-based authentication, specify your Default Application
Credentials in the `GOOGLE_CREDENTIALS` environment variable or
corresponding `google_credentials` parameter in `Config` or the client
constructors. You may provide either the path to the file containing
your credentials or the credentials themselves serialized as JSON. To
use Google impersonation, specify the service principal to impersonate
in the `DATABRICKS_GOOGLE_SERVICE_ACCOUNT` environment variable or the
corresponding `google_service_account` parameter in `Config` or the
client constructors. See
[#444](#444) for the
changes.

Bug fixes:
* Fix flask app example
([#445](#445)).
* Fix deserialization of repeated enums
([#450](#450),
[#452](#452)).
* Capture stdout and stderr separately when calling Azure CLI
([#460](#460)).

Other changes:
* Change the name of retries logger to `databricks.sdk.retries`
([#453](#453)).

API Changes:

* Added `pipeline_id` field for
`databricks.sdk.service.catalog.TableInfo`.
* Added `enable_predictive_optimization` field for
`databricks.sdk.service.catalog.UpdateCatalog` and
`databricks.sdk.service.catalog.UpdateSchema`.
* Removed `databricks.sdk.service.catalog.UpdatePredictiveOptimization`
and
`databricks.sdk.service.catalog.UpdatePredictiveOptimizationResponse`
dataclasses.
* Removed `enable_optimization()` method for
[w.metastores](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/metastores.html)
workspace-level service.
* Added `description` field for `databricks.sdk.service.jobs.CreateJob`
and `databricks.sdk.service.jobs.JobSettings`.
* Added `list_network_connectivity_configurations()` and
`list_private_endpoint_rules()` methods for
[a.network_connectivity](https://databricks-sdk-py.readthedocs.io/en/latest/account/network_connectivity.html)
account-level service.
* Added
`databricks.sdk.service.settings.ListNccAzurePrivateEndpointRulesResponse`,
`databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsRequest`,
`databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsResponse`,
and `databricks.sdk.service.settings.ListPrivateEndpointRulesRequest`
dataclasses.

Internal changes:

* Make ucws tests skipped when DATABRICKS_ACCOUNT_ID is present
([#448](#448)).

OpenAPI SHA: 22f09783eb8a84d52026f856be3b2068f9498db3, Date: 2023-11-23
Dependency updates:

 * Introduced "google-auth" dependency to support Google authentication.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pietern pietern pietern left review comments

@mgyucht mgyucht mgyucht approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@edwardfeng-db @codecov-commenter @pietern @mgyucht