diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml index 5e1e214..db640e0 100644 --- a/.github/workflows/main.yaml +++ b/.github/workflows/main.yaml @@ -26,10 +26,9 @@ jobs: # use older checkout version due to https://github.com/dawidd6/action-download-artifact/issues/261 - uses: actions/checkout@v2 - name: Import Synapse Data - run: ./import-data.sh $BRANCH $SYNAPSE_USERNAME $SYNAPSE_PASSWORD $DB_HOST $DB_USER $DB_PASS + run: ./import-data.sh $BRANCH $SYNAPSE_PASSWORD $DB_HOST $DB_USER $DB_PASS env: BRANCH: ${{ github.ref_name }} - SYNAPSE_USERNAME: ${{ secrets.SYNAPSE_USERNAME }} SYNAPSE_PASSWORD: ${{ secrets.SYNAPSE_PASSWORD }} DB_HOST: ${{ secrets.DB_HOST }} DB_USER: ${{ secrets.DB_USER }} diff --git a/README.md b/README.md index 605c340..4eab43e 100644 --- a/README.md +++ b/README.md @@ -36,11 +36,9 @@ The following secrets need to be setup in Github for the scripts to deploy datab Global secrets: -| Variable | Description | Example | -|----------------------|-----------------------------------|-----------------------------| -| SYNAPSE_USERNAME | The Synapse service user | syn-service-user | -| SYNAPSE_PASSWORD | The Synapse service user password | supersecret | - +| Variable | Description | Example | +|----------------------|----------------------------------|----------------------------------| +| SYNAPSE_PASSWORD | Synapse service user token (PAT) | glY4283tLQHZ...0eXAiOi...JKV1QiL | Context specific secrets for each environment that corresponds to a git branch (develop/staging/prod): @@ -57,8 +55,9 @@ Context specific secrets for each environment that corresponds to a git branch ( ## Self hosted runners [agora2-infra] repository deploys a bastian host in AWS for each environment which have access to -the databases. We manually configure a [Github self-hosted runner] for each bastian host, -a label is applied to each runner to match the corresponding deployment branch name (develop/staging/prod). +the databases. We manually configure a [Github self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners) +for each bastian host, a label is applied to each runner to match the corresponding git branch name (develop/staging/prod). + Each runner corresponds to an environment which corresponds to a git branch. The update is executed from these runners. When a push happens on a branch (i.e. develop), the update is executed on the `agora-bastian-develop` runner which in turn updates the development database. @@ -67,6 +66,58 @@ is executed on the `agora-bastian-develop` runner which in turn updates the deve ![alt text][self_hosted_runners] +### Setup self hosted runners + +Github self hosted runners are deployed with a [Sceptre template config file])(https://github.com/Sage-Bionetworks/agora2-infra/blob/main/config/agoradev/develop/agora-bastian.yaml). + +Self Hosted Runner setup: +* Deploy the template to the Agora AWS account. +* Login to AWS console and goto `EC2 -> select the deployed instance -> Connect -> Session Manager -> Connect` to gain ssh access to the instance. +* Follow the instructions to install the [Github self hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/adding-self-hosted-runners#adding-a-self-hosted-runner-to-a-repository). We installed it to the `/home/ssm-user/actions-runner` folder. +* Run the `config.sh` script to configure the runner. !! Important !! Make sure to set the runner `name` and `label` corresponding to the desired deployment environment (develop/staging/prod).. +```text +sh-4.2$ pwd +/home/ssm-user/actions-runner + +sh-4.2$ ./config.sh --url https://github.com/Sage-Bionetworks/agora-data-manager --token XXXXXXXXXXXXXXXXX6VLI + +-------------------------------------------------------------------------------- +| ____ _ _ _ _ _ _ _ _ | +| / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ | +| | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| | +| | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ | +| \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ | +| | +| Self-hosted runner registration | +| | +-------------------------------------------------------------------------------- + +# Authentication + + +√ Connected to GitHub + +# Runner Registration + +Enter the name of the runner group to add this runner to: [press Enter for Default] + +Enter the name of runner: [press Enter for ip-10-XXX-XXX-XXX] agora-bastian-prod + +This runner will have the following labels: 'self-hosted', 'Linux', 'X64' +Enter any additional labels (ex. label-1,label-2): [press Enter to skip] prod + +√ Runner successfully added +√ Runner connection is good + +# Runner settings + +Enter name of work folder: [press Enter for _work] + +√ Settings Saved. +``` +* Setup the [GH runner agent to run as a service](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/configuring-the-self-hosted-runner-application-as-a-service) +* Run the agent and then check the [GH Runners page](https://github.com/Sage-Bionetworks/agora-data-manager/settings/actions/runners) to make sure that the runner is in `Idle` status. + [db_update]: agora-db-update.drawio.png "update diagram" [github_secrets]: github_secrets.png "github secrets screen" [self_hosted_runners]: self-hosted-runners.png "self hosted runners" diff --git a/data-manifest.json b/data-manifest.json index 8b74810..312e106 100644 --- a/data-manifest.json +++ b/data-manifest.json @@ -1,5 +1,5 @@ { - "data-version": "68", + "data-version": "71", "data-manifest-id": "syn13363290", "team-images-id": "syn12861877" } diff --git a/import-data.sh b/import-data.sh index 767d592..7a88cba 100755 --- a/import-data.sh +++ b/import-data.sh @@ -6,11 +6,10 @@ set -e BRANCH=$1 -SYNAPSE_USERNAME=$2 -SYNAPSE_PASSWORD=$3 -DB_HOST=$4 -DB_USER=$5 -DB_PASS=$6 +SYNAPSE_PASSWORD=$2 +DB_HOST=$3 +DB_USER=$4 +DB_PASS=$5 CURRENT_DIR=$(pwd) WORKING_DIR=$CURRENT_DIR @@ -26,7 +25,7 @@ TEAM_IMAGES_ID=$(cat $WORKING_DIR/data-manifest.json | grep team-images-id | hea echo "$BRANCH branch, DATA_VERSION = $DATA_VERSION, manifest id = $DATA_MANIFEST_ID" # Download the manifest file from synapse -synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $DATA_VERSION $DATA_MANIFEST_ID +synapse -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $DATA_VERSION $DATA_MANIFEST_ID # Ensure there's a newline at the end of the manifest file; otherwise the last listed file will not be downloaded # echo >> $DATA_DIR/data_manifest.csv @@ -34,11 +33,11 @@ synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_D # Download all files referenced in the manifest from synapse cat $DATA_DIR/data_manifest.csv | tail -n +2 | while IFS=, read -r id version; do echo Downloading $id,$version - synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $version $id ; + synapse -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $version $id ; done # Download team images -synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get -r --downloadLocation $TEAM_IMAGES_DIR/ $TEAM_IMAGES_ID +synapse -p $SYNAPSE_PASSWORD get -r --downloadLocation $TEAM_IMAGES_DIR/ $TEAM_IMAGES_ID echo "Data Files: " ls -al $WORKING_DIR @@ -65,7 +64,7 @@ mongoimport -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabas mongoimport -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabase admin --collection genesbiodomains --jsonArray --drop --file $DATA_DIR/genes_biodomains.json mongoimport -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabase admin --collection biodomaininfo --jsonArray --drop --file $DATA_DIR/biodomain_info.json -mongo --host $DB_HOST -u $DB_USER -p $DB_PASS --authenticationDatabase admin $WORKING_DIR/create-indexes.js +mongosh --host $DB_HOST -u $DB_USER -p $DB_PASS --authenticationDatabase admin $WORKING_DIR/create-indexes.js pushd $TEAM_IMAGES_DIR ls -1r *.{jpg,jpeg,png} | while read x; do mongofiles -h $DB_HOST -d agora -u $DB_USER -p $DB_PASS --authenticationDatabase $DB_USER -v put $x; echo $x; done