data-playground

Data Playground

This blueprint creates a minimum viable architecture for a data experimentation project with the needed APIs enabled, VPC and Firewall set in place, BigQuery dataset, GCS bucket and an AI notebook to get started.

This is the high level diagram:

Managed resources and services

This sample creates several distinct groups of resources:

• project
• networking
• Vertex AI Workbench notebook configured with a private IP and using a dedicated Service Account
• One GCS bucket
• One BigQuery dataset

Virtual Private Cloud (VPC) design

As is often the case in real-world configurations, this blueprint accepts as input an existing Shared-VPC via the network_config variable. Make sure that 'container.googleapis.com', 'notebooks.googleapis.com' and 'servicenetworking.googleapis.com' are enabled in the VPC host project.

If the network_config variable is not provided, one VPC will be created in each project that supports network resources (load, transformation and orchestration).

Deploy your environment

We assume the identity running the following steps has the following role:

• resourcemanager.projectCreator in case a new project will be created.
• owner on the project in case you use an existing project.

Run Terraform init:

terraform init

Configure the Terraform variable in your terraform.tfvars file. You need to specify at least the following variables:

prefix = "prefix"
project_id      = "data-001"

You can run now:

terraform apply

You can now connect to the Vertex AI notbook to perform your data analysis.

Variables

name	description	type	required	default
prefix	Prefix used for resource names.	string	✓
project_id	Project id, references existing project if project_create is null.	string	✓
deletion_protection	Prevent Terraform from destroying data storage resources (storage buckets, GKE clusters, CloudSQL instances) in this blueprint. When this field is set in Terraform state, a terraform destroy or terraform apply that would delete data storage resources will fail.	bool		false
location	The location where resources will be deployed.	string		"EU"
network_config	Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values.	object({…})		null
project_create	Provide values if project creation is needed, uses existing project if null. Parent format: folders/folder_id or organizations/org_id.	object({…})		null
region	The region where resources will be deployed.	string		"europe-west1"

Outputs

name	description	sensitive
bucket	GCS Bucket URL.
dataset	GCS Bucket URL.
notebook	Vertex AI notebook details.
project	Project id.
vpc	VPC Network.

Test

module "test" {
  source     = "./fabric/blueprints/data-solutions/data-playground"
  project_id = "sampleproject"
  prefix     = "tst"
  project_create = {
    billing_account_id = "123456-123456-123456",
    parent             = "folders/467898377"
  }
}
# tftest modules=8 resources=66