-
Notifications
You must be signed in to change notification settings - Fork 9
Development Environment
The development environment is setup specifically to be
- Easy to install and tear down
- Platform independent
- Easy to configure
- Secure
- Reusable and adaptable
To do this, we use VMs we start and configure with Vagrant. We use shell scripts to provision our VM as needed so that these scripts can be run as needed on any system trying to similarly set things up. Any changes made to this scripts can be easily rerun on demand by developers as needed. In addition, vagrant provides command-line access to the VMs so that any extra developer specific configuration can be performed and needed.
The following scripts are run when a VM is being spun up
Script | Description | Runs on Vagrant | Runs on CircleCI |
---|---|---|---|
install-deps.sh |
Git Clones ISB-CGC-Common into working directory. Installs dependencies through apt-get , pip and gem . Also installs GAE and Google Cloud SDK |
☑️ | ☑️ |
create-database.sh |
Sets up the MySQL root user, password and database | ☑️ | ❌ |
database-setup.sh |
Run migrations, fills tables with any necessary data, sets up Social App login | ☑️ | ☑️ |
vagrant-server-start.sh |
Starts SASS compiler for development | ☑️ | ❌ |
vagrant-set-env.sh |
Sets vagrant environment variables for development | ☑️ | ❌ |
python-su.sh |
Sudo wrapper for python for development | ☑️ | ❌ |
gcloud_authenticate.sh |
Authenticate the system with gcloud for API use | ❌ | ☑️ |
gcloud-pull-staging-files.sh |
Pull down files needed to deploy to staging | ❌ | ☑️ |
The system uses environment variables to control the components, configure the system, and store information like database configuration securely. Environment variables can either be set on the system, or by using a .env
file in the project root folder. You can use the sample.env
file as a base for learning the format of the .env
file, but in general it is simply KEY=VALUE
, and one to a line.
Below are the environment variables and what they are needed for.
Variable | Description |
---|---|
MYSQL_ROOT_USER |
Root user. Used during setup of system in scripts |
MYSQL_ROOT_PASSWORD |
Root Password. Used during setup of system in scripts |
GCLOUD_PROJECT_ID |
Project ID as defined by Google Cloud. Used during both setup and run of the system |
DJANGO_SECRET_KEY |
A secret key for Django to use. Used while running |
BASE_URL |
Base URL for the deployed Web Application, eg. http://localhost:8080 or https://isb-cgc.appspot.com |
COHORT_DATASET_ID |
BigQuery dataset for storing cohorts. For the Developer Environments this is dev_deployment_cohorts . Each tier has its own dataset for the deployment cohort dataset. |
BIGQUERY_COHORT_TABLE_ID |
BigQuery table for storing cohorts. For Developer Environments this will be a personal table under the dev_deployment_cohorts dataset. Each deployed tier has its own table within its BQ cohort dataset. |
API_URL |
API URL for the system, if being used, eg. https://api-dot-isb-cgc.appspot.com |
DATABASE_HOST |
IP or Hostname for the Database |
DATABASE_NAME |
Name of the database to store data |
DATABASE_USER |
User to use to login to the database |
DATABASE_PASSWORD |
Database user's password |
SUPERUSER_PASSWORD |
Default password for the superuser. Only used for setup |
GOOGLE_CLIENT_ID |
Google Cloud Credentials Client ID |
GOOGLE_CLIENT_SECRET |
Google Cloud Credentials Client Secret |
GOOGLE_APPLICATION_CREDENTIALS |
Filepath location of the privatekey.json file |
SERVICE_ACCOUNT_BLACKLIST_PATH |
List of ISB-CGC Service Accounts, JSON file |
GOOGLE_ORG_WHITELIST_PATH |
List of Google Organizations allowed as parents of GCP Projects, JSON file |
MANAGED_SERVICE_ACCOUNTS_PATH |
List of Service Accounts which are Google Managed, JSON file |
CAMIC_VIEWER |
URL to the caMicroscope Viewer Proxy |
IMG_THUMBS_URL |
URL to the caMicroscope thumbnails directory |
Variable | Description |
---|---|
DATASET_CONFIGURATION_PATH |
List of Datasets and the locations of their whitelists and logs, JSON file |
Variable | Description |
---|---|
IS_DEV |
Flag identifying the environment as a developer environment and not a cloud-deployed system. |
DEBUG_TOOLBAR |
Flag indicating if the Django Debug Toolbar should be installed. Only valid for developer environments. |
Variable | Description | Default |
---|---|---|
BIGQUERY_PROJECT_ID |
Project ID for the Google Big Query data | GCLOUD_PROJECT_ID |
SUPERUSER_USERNAME |
Username of the superuser | isb |
SUPERUSER_PASSWORD |
Password of the superuser | -- |
DATABASE_ENGINE |
Engine to use for the DB connection | django.db.backends.mysql |
DATABASE_PORT |
Port on the DATABASE_HOST to connect to |
3306 |
If you want to setup a configuration in the system of something, you should do so through environment variables. Names should be descriptive and unique enough to distinguish settings from one another. Also, whenever possible, Environment variables should be used in favor of configuration that duplicates types of data. For example, you should avoid have DATABASE_HOST
and CLOUD_DATABASE_HOST
unless both will be need simultaneously. If you only need one for a particular environment, it should be changed via the environment variables.
Any settings that are commonly needed for development purposes should be included in the sample.env
file in the root.
The VM is running Ubuntu 14.04 and forwards ports 8000 and 8080. It also add the project directory and the projects parent directory to the VM as shared folders so that development can be done seamlessly.
Log files will be placed in the parent directory of the root machine. The following files will have the following output.
File | Description |
---|---|
outputSASS.log |
SASS output logs |
errorSASS.log |
SASS compile errors |
This project depends on both ISB-CGC-Common and ISB-CGC-API. These projects should be cloned next to the ISB-CGC-Webapp repository. In the Vagrant file, these directories will be mapped to directories inside the ISB-CGC-Webapp directory.
In PyCharm, you will also need to add those cloned repositories as source folders for this project.