Accessing storage with shared integration Databricks?

[baatch]

What is the recommended way to access the raw, enrich/cur, workspace data lakes from the shared integration Databricks?

Using a service principal and mount point?
If yes, where should service principal credential be stored?
Hive Key Vault, ADF Key Vault, Seperate?

[marvinbuss]

Hi @baatch,
The recommended approach would be to use Credential Passthrough (OAuth 2.0). For automated workflows, you should use a Service Principle and store the secrets (clientId, clientSecret) in a Key Vault backed secret store.
The respective integration and product teams can use their own Key Vault within their RG to store the secrets.

[baatch]

@marvinbuss so is this the correct understanding of how to use the shared integration databricks workspace?:

• create an service principal and store secret in the key vault that stores the hive mysql credential
• configure credential passthrough on workspace and cluster
• give service principal permissions on data lake folders in each storage account (raw, enrcur, work)
• in notebook access service principal secret throught key vault and read and write to data lake with abfs:// paths and not use mount points?

should this be included in the template solution?

[marvinbuss]

The shared integration Databricks workspace should be used for the enrichment and curation of datasets that are integrated/ingested into the Data Landing Zone. In this workspace you will mostly find Jobs running in an automated fashion using Standard Clusters (one per Job). These Jobs are taking the data from the raw layer and save the processed datasets onto the raw and enriched layer. For data access, these Clusters should rely on Service Principles and Credential Passthrough (OAuth2.0). This must be configured accordingly on the workspace and cluster.

Enterprise-Scale Analytics is using and identity based authentication model. Regardless of the compute engine, access is controlled on the storage layer, which simplifies the number of points where access needs to be controlled. The respective service principle needs to be granted Reader access on the raw layer and read&write access on the enriched&cureated layer, where it will save the processed version, which will then be shared with the Data Product teams within the Data Landing Zones.

We would like to further automate some of the automation wrt Databricks, but sadly, all users are blocked today because of this: databricks/databricks-cli#338 (see also here #14). Once this would get fixed, we would immediately automate additional steps within the Data Landing Zone and within the Databricks workspaces. However, we want to rely on Key Vault backed secret scopes and the creation of them inside a workspace cannot be automated today. The API does not support the on-behalf-of (OBO) workflow.

Service Principles should not be shared due to access restrictions and to follow the least privilege principle. If you use the Key Vault where the Hive DB connection details are stored, you would risk that other teams may also be able to access the same service principle details as they need all access to that secret scope to connect with their cluster to the Hive metastore. This is the case, because within a Key Vault you can only grant access to all or no secrets within the service instance. The service principle details (clientId, clientSecret, tenantId) should be optimally stored in the Key Vault of the Data Integration team. Key Vaults do not create any additional cost and hence I would store them seperate.

[baatch]

@marvinbuss thanks for the good explanation!