Merge pull request #130 from Sage-Bionetworks/staging

AG-1517: Deploy mv71 to production for 3.5.0 release

.github/workflows/main.yaml

@@ -26,10 +26,9 @@ jobs:
       # use older checkout version due to https://github.com/dawidd6/action-download-artifact/issues/261
       - uses: actions/checkout@v2
       - name: Import Synapse Data
-        run: ./import-data.sh $BRANCH $SYNAPSE_USERNAME $SYNAPSE_PASSWORD $DB_HOST $DB_USER $DB_PASS
+        run: ./import-data.sh $BRANCH $SYNAPSE_PASSWORD $DB_HOST $DB_USER $DB_PASS
         env:
           BRANCH: ${{ github.ref_name }}
-          SYNAPSE_USERNAME: ${{ secrets.SYNAPSE_USERNAME }}
           SYNAPSE_PASSWORD: ${{ secrets.SYNAPSE_PASSWORD }}
           DB_HOST: ${{ secrets.DB_HOST }}
           DB_USER: ${{ secrets.DB_USER }}

README.md

@@ -36,11 +36,9 @@ The following secrets need to be setup in Github for the scripts to deploy datab
 
 Global secrets:
 
-| Variable             | Description                       | Example                     |
-|----------------------|-----------------------------------|-----------------------------|
-| SYNAPSE_USERNAME     | The Synapse service user          | syn-service-user            |
-| SYNAPSE_PASSWORD     | The Synapse service user password | supersecret                 |
-
+| Variable             | Description                      | Example                          |
+|----------------------|----------------------------------|----------------------------------|
+| SYNAPSE_PASSWORD     | Synapse service user token (PAT) | glY4283tLQHZ...0eXAiOi...JKV1QiL |
 
 Context specific secrets for each environment that corresponds to a git branch (develop/staging/prod):
 
@@ -57,8 +55,9 @@ Context specific secrets for each environment that corresponds to a git branch (
 ## Self hosted runners
 
 [agora2-infra] repository deploys a bastian host in AWS for each environment which have access to
-the databases.  We manually configure a [Github self-hosted runner] for each bastian host,
-a label is applied to each runner to match the corresponding deployment branch name (develop/staging/prod).
+the databases.  We manually configure a [Github self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners)
+for each bastian host, a label is applied to each runner to match the corresponding git branch name (develop/staging/prod).
+
 Each runner corresponds to an environment which corresponds to a git branch. The update is
 executed from these runners.  When a push happens on a branch (i.e. develop), the update
 is executed on the `agora-bastian-develop` runner which in turn updates the development database.
@@ -67,6 +66,58 @@ is executed on the `agora-bastian-develop` runner which in turn updates the deve
 ![alt text][self_hosted_runners]
 
 
+### Setup self hosted runners
+
+Github self hosted runners are deployed with a [Sceptre template config file])(https://github.com/Sage-Bionetworks/agora2-infra/blob/main/config/agoradev/develop/agora-bastian.yaml).
+
+Self Hosted Runner setup:
+* Deploy the template to the Agora AWS account.
+* Login to AWS console and goto `EC2 -> select the deployed instance -> Connect -> Session Manager -> Connect` to gain ssh access to the instance.
+* Follow the instructions to install the [Github self hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository).  We installed it to the `/home/ssm-user/actions-runner` folder.
+* Run the `config.sh` script to configure the runner.  !! Important !! Make sure to set the runner `name` and `label` corresponding to the desired deployment environment (develop/staging/prod)..
+```text
+sh-4.2$ pwd
+/home/ssm-user/actions-runner
+
+sh-4.2$ ./config.sh --url https://github.com/Sage-Bionetworks/agora-data-manager --token XXXXXXXXXXXXXXXXX6VLI
+
+--------------------------------------------------------------------------------
+|        ____ _ _   _   _       _          _        _   _                      |
+|       / ___(_) |_| | | |_   _| |__      / \   ___| |_(_) ___  _ __  ___      |
+|      | |  _| | __| |_| | | | | '_ \    / _ \ / __| __| |/ _ \| '_ \/ __|     |
+|      | |_| | | |_|  _  | |_| | |_) |  / ___ \ (__| |_| | (_) | | | \__ \     |
+|       \____|_|\__|_| |_|\__,_|_.__/  /_/   \_\___|\__|_|\___/|_| |_|___/     |
+|                                                                              |
+|                       Self-hosted runner registration                        |
+|                                                                              |
+--------------------------------------------------------------------------------
+
+# Authentication
+
+
+√ Connected to GitHub
+
+# Runner Registration
+
+Enter the name of the runner group to add this runner to: [press Enter for Default]
+
+Enter the name of runner: [press Enter for ip-10-XXX-XXX-XXX] agora-bastian-prod
+
+This runner will have the following labels: 'self-hosted', 'Linux', 'X64'
+Enter any additional labels (ex. label-1,label-2): [press Enter to skip] prod
+
+√ Runner successfully added
+√ Runner connection is good
+
+# Runner settings
+
+Enter name of work folder: [press Enter for _work]
+
+√ Settings Saved.
+```
+* Setup the [GH runner agent to run as a service](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/configuring-the-self-hosted-runner-application-as-a-service)
+* Run the agent and then check the [GH Runners page](https://github.com/Sage-Bionetworks/agora-data-manager/settings/actions/runners) to make sure that the runner is in `Idle` status.
+
 [db_update]: agora-db-update.drawio.png "update diagram"
 [github_secrets]: github_secrets.png "github secrets screen"
 [self_hosted_runners]: self-hosted-runners.png "self hosted runners"

data-manifest.json

@@ -1,5 +1,5 @@
 {
-    "data-version": "68",
+    "data-version": "71",
     "data-manifest-id": "syn13363290",
     "team-images-id": "syn12861877"
 }

import-data.sh

@@ -6,11 +6,10 @@
 set -e
 
 BRANCH=$1
-SYNAPSE_USERNAME=$2
-SYNAPSE_PASSWORD=$3
-DB_HOST=$4
-DB_USER=$5
-DB_PASS=$6
+SYNAPSE_PASSWORD=$2
+DB_HOST=$3
+DB_USER=$4
+DB_PASS=$5
 
 CURRENT_DIR=$(pwd)
 WORKING_DIR=$CURRENT_DIR
@@ -26,19 +25,19 @@ TEAM_IMAGES_ID=$(cat $WORKING_DIR/data-manifest.json | grep team-images-id | hea
 echo "$BRANCH branch, DATA_VERSION = $DATA_VERSION, manifest id = $DATA_MANIFEST_ID"
 
 # Download the manifest file from synapse
-synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $DATA_VERSION $DATA_MANIFEST_ID
+synapse -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $DATA_VERSION $DATA_MANIFEST_ID
 
 # Ensure there's a newline at the end of the manifest file; otherwise the last listed file will not be downloaded
 # echo >> $DATA_DIR/data_manifest.csv
 
 # Download all files referenced in the manifest from synapse
 cat $DATA_DIR/data_manifest.csv | tail -n +2 | while IFS=, read -r id version; do
   echo Downloading $id,$version
-    synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $version $id ;
+    synapse -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $version $id ;
   done
 
 # Download team images
-synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get -r --downloadLocation $TEAM_IMAGES_DIR/ $TEAM_IMAGES_ID
+synapse -p $SYNAPSE_PASSWORD get -r --downloadLocation $TEAM_IMAGES_DIR/ $TEAM_IMAGES_ID
 
 echo "Data Files: "
 ls -al $WORKING_DIR
@@ -65,7 +64,7 @@ mongoimport -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabas
 mongoimport -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabase admin --collection genesbiodomains --jsonArray --drop --file $DATA_DIR/genes_biodomains.json
 mongoimport -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabase admin --collection biodomaininfo --jsonArray --drop --file $DATA_DIR/biodomain_info.json
 
-mongo --host $DB_HOST -u $DB_USER -p $DB_PASS --authenticationDatabase admin $WORKING_DIR/create-indexes.js
+mongosh --host $DB_HOST -u $DB_USER -p $DB_PASS --authenticationDatabase admin $WORKING_DIR/create-indexes.js
 
 pushd $TEAM_IMAGES_DIR
 ls -1r *.{jpg,jpeg,png} | while read x; do mongofiles -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabase $DB_USER -v put $x; echo $x; done