Update README regarding generation of model from Google Sheet

[turbomam]

See cancerDHC/ccdhmodel#96 (review)

[turbomam]

• cloned repo into fresh directory
• switched to branch merge-pipenv
• make runs to completion but complains about the documentation structure:

INFO - Cleaning site directory
INFO - Building documentation to directory: /Users/MAM/Documents/gitrepos/cr2/ccdhmodel/docs
INFO - The following pages exist in the docs directory, but are not included in the "nav" configuration:
- AlcoholExposureObservation.md
...
- treatment__treatment_type.md
INFO - Documentation built in 7.12 seconds

• make test runs to completion with 1 pass and 0 failures
• make generate-model with no preparation on my part

% make generate-model
pipenv install
Installing dependencies from Pipfile.lock (60b98b)...
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:12
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
CDM_GOOGLE_SHEET_ID=1oWS7cao-fgz2MKWtyr8h2dEL9unX__0bJrWKv6mQmM4 PIPENV_IGNORE_INSTALLED=1 pipenv run python vendor/sheet2linkml/sheet2linkml.py --output model/schema/crdch_model.yaml
Traceback (most recent call last):
File "vendor/sheet2linkml/sheet2linkml.py", line 6, in
from sheet2linkml import cli
File "/Users/MAM/Documents/gitrepos/cr2/ccdhmodel/vendor/sheet2linkml/sheet2linkml/cli.py", line 16, in
from sheet2linkml.source.gsheetmodel.gsheetmodel import GSheetModel
File "/Users/MAM/Documents/gitrepos/cr2/ccdhmodel/vendor/sheet2linkml/sheet2linkml/source/gsheetmodel/gsheetmodel.py", line 2, in
from sheet2linkml.source.gsheetmodel.mappings import Mappings
File "/Users/MAM/Documents/gitrepos/cr2/ccdhmodel/vendor/sheet2linkml/sheet2linkml/source/gsheetmodel/mappings.py", line 20, in
class Mappings:
File "/Users/MAM/Documents/gitrepos/cr2/ccdhmodel/vendor/sheet2linkml/sheet2linkml/source/gsheetmodel/mappings.py", line 72, in Mappings
) -> list[Mapping]:
TypeError: 'type' object is not subscriptable
make: *** [generate-model] Error 1

• deleted Google Cloud projects that In ahd previously used with this repo
• visited the Google Cloud Platform (GCP) at https://console.cloud.google.com/
• hamburger menu in upper left -> IAM & Admin -> Create a Project
  • or use the "Search Products and Services" text box at the top of the web page
• Enter a project name. Select an organization and location if relevant. I did "ccdh-model-gen-20210908", "lbl.gov" and "lbl.gov"
• GCP hamburger menu -> APIs & Services -> Library
• Enter into "Search for APIs & Services" test box at top: "google drive api" and hit enter
• Click the tile for the "Google Drive API"
• Click the "Enable" button
• Repeat the previous steps (from GCP hamburger menu -> APIs & Services -> Library) for the Google Sheets API
• I am not sure which of the two are actually required for this repo
• GCP hamburger menu -> APIs & Services -> Credentials
• Click "Configure Consent Screen" button in upper right
• Click "External" User Type radio button and then click "CREATE" button
• Fill in the required fields (with red asterisks)
  • I entered "ccdhmodel generator" for "App information"->"App name"
  • "[email protected]" for "App information"->"User support email"
  • "https://github.com/cancerDHC/ccdhmodel" for (optional) "App domain"->"Application home page"
  • "[email protected]" for "Developer contact information"->"Email address"
• Click "SAVE AND CONTINUE"
• Click "SAVE AND CONTINUE" again on "Scopes" screen
• Click "SAVE AND CONTINUE" again on "Test users" screen
• Click "BACK TO DASHBOARD" on "Summary" screen
• Click "PUBLISH APP" on "OAuth consent screen" screen->"Publishing status"
• GCP hamburger menu -> APIs & Services -> Credentials
  • Click the "+ CREATE CREDENTIALS" button in the upper left
  • Select "OAuth Client ID"
  • Select "Desktop App" from the "App type" pulldown menu
  • Enter an app name in the"Name" box. I used "ccdh-model-gen-20210908"
  • Click the "CREATE" button
  • Click the "DOWNLOAD JSON" button on the "OAuth client created" dialogue box
  • Note the following warning:

OAuth is limited to 100 sensitive scope logins until the OAuth consent screen is verified. This may require a verification process that can take several days.

• Click "OK" to dismiss the "OAuth client created" dialogue box
• Find the file that you just downloaded. Depending on the web browser you're using, t might be in your Downloads directory, if you didn't specify otherwise.
• The file will probably have a name that follows this pattern: client_secret_*.apps.googleusercontent.com.json.
• Copy that file to the root of your local ccdhmodel directory and give it this name: google_api_credentials.json
• run make generate-model in the root of your local ccdhmodel directory

still getting TypeError: 'type' object is not subscriptable as above

also copied google_api_credentials.json into vendor/sheet2linkml/

[turbomam]

Cheating by going to a previous ccdhmodel folder

google_api_credentials.json is in repo root

% make generate-model
pipenv install
Installing dependencies from Pipfile.lock (60b98b)...
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:11
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
CDM_GOOGLE_SHEET_ID=1oWS7cao-fgz2MKWtyr8h2dEL9unX__0bJrWKv6mQmM4 PIPENV_IGNORE_INSTALLED=1 pipenv run python vendor/sheet2linkml/sheet2linkml.py --output model/schema/crdch_model.yaml
Loading .env environment variables...
Please go to this URL and finish the authentication flow: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=25191608779-77qsof3vglumvlbn2d9n35p55ntncb8e.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.metadata.readonly&state=Gd75o3pF7DyO48W90x1tKgg66OPcTD&prompt=consent&access_type=offline
Enter the authorization code:

• paste link into web browser (Chrome)
• select or enter user (should be the same as the user who was logged into the GCP above)
• review warnings and click "Allow" in lower right
• copy the code on the next screen. (There's a handy copy button that looks like one page of paper stacked on top of another).
• paste back into the console at the Enter the authorization code prompt and hit enter
• generation of model from Google Sheet commences

[turbomam]

See cancerDHC/ccdhmodel#96 (comment)

I don't see this. On the "OAuth consent screen" tab on your GCI APIs and Services page, could you please make sure that your app has a publishing status of "Testing" and that your e-mail address has been added to the list of Test Users? I think that's the way to suppress this warning.

• set to GCP publication status to testing
• added [email protected] as a test user
• started fresh repo
• git checkout merge-pipenv
• GCP hamburger menu -> APIs & Services -> Credentials -> click on relevant "OAuth 2.0 Client IDs" -> Download JSON
• copy downloaded file ~/Downloads/client_secret_*.apps.googleusercontent.com.json to ./google_api_credentials.json

Maybe not necessary any more?

ccdhmodel % cat > .env
CDM_GOOGLE_SHEET_ID=1oWS7cao-fgz2MKWtyr8h2dEL9unX__0bJrWKv6mQmM4
ccdhmodel % 

• make generate-model

Pipenv setup went well

CDM_GOOGLE_SHEET_ID=1oWS7cao-fgz2MKWtyr8h2dEL9unX__0bJrWKv6mQmM4 PIPENV_IGNORE_INSTALLED=1 pipenv run python vendor/sheet2linkml/sheet2linkml.py --output model/schema/crdch_model.yaml
Loading .env environment variables...
Please go to this URL and finish the authentication flow: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=25191608779-77qsof3vglumvlbn2d9n35p55ntncb8e.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fspreadsheets.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.metadata.readonly&state=MdfyOWgvngrocYbtHRLEam6T3m7Ket&prompt=consent&access_type=offline
Enter the authorization code: 

Pasted link into browser. No "Google hasn’t verified this app" warning!

Confirmed that I want to authenticate as [email protected]. Approved the data access requests. Copied the provided code and pasted back into terminal. Hit enter. Model generation goes to completion.

[gaurav]

Odd. Too bad we couldn't get to the bottom of what was going on. Still, it's nice to have this documented in the issues!

[turbomam]

Why odd? I was trying to say that everything works and makes more sense to me now. Maybe that's not clear amidst all of the verbosity.

[gaurav]

Ah, I thought the problem had "fixed itself", which is always annoying :). Looking more closely, it looks like maybe "set to GCP publication status to testing" worked!

[sujaypatil96]

I think the contents on the README file for this repo are fairly succinct, and that this issue is ready to be closed.