Skip to content

Latest commit

 

History

History
49 lines (35 loc) · 959 Bytes

README.md

File metadata and controls

49 lines (35 loc) · 959 Bytes

Big Data LDN Digitransit demo

Architecture diagram

Instructions

  1. Create env.sh containing your Aiven API token
cat > env.sh <<EOF
export TF_VAR_aiven_api_token=<your api key>
EOF
  1. Create infrastructure

Note: Update terraform.tfvars with your Aiven project name first.

$ source env.sh
$ terraform init
$ terraform plan
$ terraform apply
  1. Create Python 3 virtualenv and install dependencies
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
  1. Load reference data into PostgreSQL (requires psql to be installed)
$ avn service cli bigdataldn-demo-postgres
defaultdb=> \i scripts/create_postgres_schema.sql; 
  1. Create ClickHouse materialized view using SQL in scripts/create_clickhouse_view.sql

  2. Prepare environment variables for Python Notebook

$ cd notebooks
$ bash env.sh

Start your prefererred Notebook environment