Skip to content
This repository has been archived by the owner on Jul 24, 2023. It is now read-only.

Commit

Permalink
Add file re-upload instructions (#21)
Browse files Browse the repository at this point in the history
* Update programmatic-upload-aws-cli.md

* Update programmatic-upload-gcp-cli.md

* Update programmatic-upload-gcp-cli.md

* Edit with notes from Yooree
  • Loading branch information
clarisse-lau authored May 5, 2022
1 parent b28a12f commit 435643d
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 5 deletions.
10 changes: 7 additions & 3 deletions programmatic-upload-aws-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,12 @@ do
done
```

Once your data is in the bucket, it will automatically be synced with your center’s Synapse project. This process can take anywhere from a few minutes to up to a day depending on the size of your data. Once it is present on Synapse, you can proceed to annotate your metadata.
Once your data is in the bucket, it will automatically be synced with your center’s Synapse project. This process can take a few minutes depending on the size of your data. Once it is present on Synapse, you can proceed to annotate your metadata.

**Note**: If you would like to make changes to your data, please do so directly from the S3 bucket and not from the Synapse web or programmatic clients. Changes made to the bucket will automatically be updated on the Synapse project.
### Update Existing Data:
If you have made changes to a file already in the bucket and would like to upload the new version, please re-upload the modified file to the same S3 key (overwrite the existing file). As with all S3 uploads, **ensure that the md5 is included as a metadata flag for larger files**. The change will be reflected on Synapse as a new version of the existing Synapse entity.

**Troubleshooting**: If you encounter ‘AccessDenied’ errors, run `aws sts get-caller-identity` to confirm your IAM identity matches that provided to the DCC. If issues persist or for any other questions, open a support ticket [here](https://sagebionetworks.jira.com/servicedesk/customer/portal/1).
**Note**: Please refrain from moving data in the bucket or adding new versions of data to a separate folder. Doing so will create new Synapse entities for your files, which will require modifying file paths and Synapse IDs in your metadata manifest.

### Troubleshooting:
If you encounter `AccessDenied` errors, run `aws sts get-caller-identity` to confirm your IAM identity matches that provided to the DCC. If issues persist or for any other questions, open a support ticket with the [HTAN Help Desk](https://sagebionetworks.jira.com/servicedesk/customer/portal/1).
7 changes: 5 additions & 2 deletions programmatic-upload-gcp-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ For large files, parallel composite uploads may be enabled for faster upload spe

`gsutil -h x-goog-meta-content-md5:<md5> cp <file> gs://<MyBucket>/<MyFolder>/`

Once your data is in the bucket, it will automatically be synced with your center’s Synapse project. This process can take anywhere from a few minutes to up to a day depending on the size of your data. Once it is present on Synapse, you can proceed to annotate your metadata.
Once your data is in the bucket, it will automatically be synced with your center’s Synapse project. This process can take a few minutes depending on the size of your data. Once it is present on Synapse, you can proceed to annotate your metadata.

**Note**: If you would like to make changes to your data, please do so directly from the Google Storage Bucket and not from the Synapse web or programmatic clients. Changes made to the bucket will automatically be updated on the Synapse project.
### Update Existing Data:
If you have made changes to a file already in the bucket and would like to upload the new version, please reach out to your DCC liaison. The DCC will grant you access to overwrite your previously added file. You may then re-upload the modified file to the same GCS key. The change will be reflected on Synapse as a new version of the existing Synapse entity.

**Note**: Please refrain from moving data in the bucket or adding new versions of data to a separate folder. Doing so will create new Synapse entities for your files, which will require modifying file paths and Synapse IDs in your metadata manifest.

0 comments on commit 435643d

Please sign in to comment.