Development Environment

The development environment is setup specifically to be

• Easy to install and tear down
• Platform independent
• Easy to configure
• Secure
• Reusable and adaptable

To do this, we use VMs we start and configure with Vagrant. We use shell scripts to provision our VM as needed so that these scripts can be run as needed on any system trying to similarly set things up. Any changes made to this scripts can be easily rerun on demand by developers as needed. In addition, vagrant provides command-line access to the VMs so that any extra developer specific configuration can be performed and needed.

Scripts

The following scripts are run when a VM is being spun up

Script	Description	Runs on Vagrant	Runs on CircleCI
install-deps.sh	Git Clones ISB-CGC-Common into working directory. Installs dependencies through apt-get, pip and gem. Also installs GAE and Google Cloud SDK	☑️	☑️
create-database.sh	Sets up the MySQL root user, password and database	☑️	❌
database-setup.sh	Run migrations, fills tables with any necessary data, sets up Social App login	☑️	☑️
vagrant-server-start.sh	Starts SASS compiler for development	☑️	❌
vagrant-set-env.sh	Sets vagrant environment variables for development	☑️	❌
python-su.sh	Sudo wrapper for python for development	☑️	❌
gcloud_authenticate.sh	Authenticate the system with gcloud for API use	❌	☑️
gcloud-pull-staging-files.sh	Pull down files needed to deploy to staging	❌	☑️

Environment Variables

The system uses environment variables to control the components, configure the system, and store information like database configuration securely. Environment variables can either be set on the system, or by using a .env file in the project root folder. You can use the sample.env file as a base for learning the format of the .env file, but in general it is simply KEY=VALUE, and one to a line.

Below are the environment variables and what they are needed for.

Common Environment Variables

Variable	Description
MYSQL_ROOT_USER	Root user. Used during setup of system in scripts
MYSQL_ROOT_PASSWORD	Root Password. Used during setup of system in scripts
GCLOUD_PROJECT_ID	Project ID as defined by Google Cloud. Used during both setup and run of the system
DJANGO_SECRET_KEY	A secret key for Django to use. Used while running
BASE_URL	Base URL for the deployed Web Application, eg. http://localhost:8080 or https://isb-cgc.appspot.com
COHORT_DATASET_ID	BigQuery dataset for storing cohorts. For the Developer Environments this is dev_deployment_cohorts. Each tier has its own dataset for the deployment cohort dataset.
BIGQUERY_COHORT_TABLE_ID	BigQuery table for storing cohorts. For Developer Environments this will be a personal table under the dev_deployment_cohorts dataset. Each deployed tier has its own table within its BQ cohort dataset.
API_URL	API URL for the system, if being used, eg. https://api-dot-isb-cgc.appspot.com
DATABASE_HOST	IP or Hostname for the Database
DATABASE_NAME	Name of the database to store data
DATABASE_USER	User to use to login to the database
DATABASE_PASSWORD	Database user's password
SUPERUSER_PASSWORD	Default password for the superuser. Only used for setup
GOOGLE_CLIENT_ID	Google Cloud Credentials Client ID
GOOGLE_CLIENT_SECRET	Google Cloud Credentials Client Secret
GOOGLE_APPLICATION_CREDENTIALS	Filepath location of the privatekey.json file
SERVICE_ACCOUNT_BLACKLIST_PATH	List of ISB-CGC Service Accounts, JSON file
GOOGLE_ORG_WHITELIST_PATH	List of Google Organizations allowed as parents of GCP Projects, JSON file
MANAGED_SERVICE_ACCOUNTS_PATH	List of Service Accounts which are Google Managed, JSON file
CAMIC_VIEWER	URL to the caMicroscope Viewer Proxy
IMG_THUMBS_URL	URL to the caMicroscope thumbnails directory

Deployed Environment Variables

Variable	Description
DATASET_CONFIGURATION_PATH	List of Datasets and the locations of their whitelists and logs, JSON file

Developer Environment Variables

Variable	Description
IS_DEV	Flag identifying the environment as a developer environment and not a cloud-deployed system.
DEBUG_TOOLBAR	Flag indicating if the Django Debug Toolbar should be installed. Only valid for developer environments.

Other Environment Variables

Variable	Description	Default
BIGQUERY_PROJECT_ID	Project ID for the Google Big Query data	GCLOUD_PROJECT_ID
SUPERUSER_USERNAME	Username of the superuser	isb
SUPERUSER_PASSWORD	Password of the superuser	--
DATABASE_ENGINE	Engine to use for the DB connection	django.db.backends.mysql
DATABASE_PORT	Port on the DATABASE_HOST to connect to	3306

Adding Environment Variables

If you want to setup a configuration in the system of something, you should do so through environment variables. Names should be descriptive and unique enough to distinguish settings from one another. Also, whenever possible, Environment variables should be used in favor of configuration that duplicates types of data. For example, you should avoid have DATABASE_HOST and CLOUD_DATABASE_HOST unless both will be need simultaneously. If you only need one for a particular environment, it should be changed via the environment variables.

Any settings that are commonly needed for development purposes should be included in the sample.env file in the root.

VM Environment

The VM is running Ubuntu 14.04 and forwards ports 8000 and 8080. It also add the project directory and the projects parent directory to the VM as shared folders so that development can be done seamlessly.

Log Files

Log files will be placed in the parent directory of the root machine. The following files will have the following output.

File	Description
outputSASS.log	SASS output logs
errorSASS.log	SASS compile errors

Working in PyCharm With Shared Repositories

This project depends on both ISB-CGC-Common and ISB-CGC-API. These projects should be cloned next to the ISB-CGC-Webapp repository. In the Vagrant file, these directories will be mapped to directories inside the ISB-CGC-Webapp directory.

In PyCharm, you will also need to add those cloned repositories as source folders for this project.