Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Broke again? GarminException: Did not find the display name in the profile page. #95

Closed
app4g opened this issue Sep 26, 2023 · 42 comments
Closed

Broke again? GarminException: Did not find the display name in the profile page. #95

app4g opened this issue Sep 26, 2023 · 42 comments

Comments

@app4g
Copy link

app4g commented Sep 26, 2023

Seems to be able to get the Ticket, but after that, it's not getting anything.

➜ garmin-connect-export-master python gcexport.py --count 1
Welcome to Garmin Connect Exporter!
[WARNING] Output directory ./2023-09-26_garmin_connect_export already exists. Will skip already-downloaded files and append to the CSV file.
Username: [email protected]
Password:
Connecting to Garmin Connect... Done.
Requesting Login ticket... Done. Ticket=ST-0426532-gzLYoa1LJbUhxVun3keA-cas
Authenticating... Done.
Getting display name...Traceback (most recent call last):
File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 1349, in
main(sys.argv)
File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 1301, in main
userstats_json = fetch_userstats(args)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 952, in fetch_userstats
display_name = extract_display_name(profile_page)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/username/Downloads/garmin-connect-export-master/gcexport.py", line 977, in extract_display_name
raise GarminException('Did not find the display name in the profile page.')
GarminException: Did not find the display name in the profile page.
@sgowtham
Copy link

sgowtham commented Sep 26, 2023 via email

@app4g
Copy link
Author

app4g commented Sep 26, 2023

I've tried a few different packages but those that I can get to work (as in compile in my machine) are not able to get anything from Garmin Servers as well. I was inspecting the login headers via chrome developer mode and I don't seem to see any "nk":"NT" headers which I believe I saw before (too bad I didn't take a screenshot of it previously to compare it to current)

@SimonBaars
Copy link
Contributor

SimonBaars commented Sep 26, 2023 •
edited
Loading

Hi, I started investigating this, and it appears they moved the call to get the displayName to a separate JSON call on the member page. The displayName field embedded in the page now contains an id, which is the id that call be used to do a separate JSON call to get the name. This is the precise call that now gets the name (as fullName field):

curl 'https://connect.garmin.com/connection-service/connection/connections/pagination/[id]?start=1&limit=24&displayMutedStatus=false&_=[number]' \
  -H 'authority: connect.garmin.com' \
  -H 'accept: application/json, text/javascript, */*; q=0.01' \
  -H 'accept-language: en-US,en;q=0.9,nl-NL;q=0.8,nl;q=0.7' \
  -H 'authorization: Bearer [token]' \
  -H 'baggage: sentry-environment=prod,sentry-release=connect%404.71.149,sentry-public_key=[number],sentry-trace_id=[number],sentry-sample_rate=1' \
  -H 'cookie: __cflb=[number]; _cfuvid=[number]; GarminUserPrefs=en-US; TAsessionID=[number]|NEW; notice_behavior=implied,eu; __cfruid=[number]; GARMIN-SSO=1; GARMIN-SSO-CUST-GUID=[number]; SESSIONID=[number]; JWT_FGP=[number]; SameSite=None; ADRUM_BTa=[number]; ADRUM_BT1=[number]' \
  -H 'di-backend: connectapi.garmin.com' \
  -H 'nk: NT' \
  -H 'referer: https://connect.garmin.com/modern/profile' \
  -H 'sec-ch-ua: "Google Chrome";v="117", "Not;A=Brand";v="8", "Chromium";v="117"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Linux"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sentry-trace: [number]' \
  -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36' \
  -H 'x-app-ver: 4.71.1.3' \
  -H 'x-lang: en-US' \
  -H 'x-requested-with: XMLHttpRequest' \
  --compressed

Since the Garmin website uses Cloudflare, we'll have to obtain a Cloudflare token to script this request as a replacement of the current member page request.

I'm working on this.

@app4g
Copy link
Author

app4g commented Sep 26, 2023

Wow.. how did you managed to locate all these items? I am only using Chrome's Developer Tools to look at the returned headers and such and didn't gleam all these info.

👍

@SimonBaars
Copy link
Contributor

So I figured out the root cause behind all of this: authentication is broken. All web calls done by this script fail, even though through CURL they work just fine. It might be that they introduced Cloudflare, or there's a different issue with auth.

@app4g
Copy link
Author

app4g commented Sep 26, 2023

CloudFlare has been there for some time already I believe. Either that or it's on for some user and not for some other users.

@SimonBaars
Copy link
Contributor

Yeah, I'm not yet sure if Cloudflare is part of the problem. With CURL, my requests succeed. I'm now comparing the headers to see if there's one they may have added.

@SimonBaars
Copy link
Contributor

Just to update on the progress: the old login page seems to set a different session cookie (called SESSION, but we need a different one called SESSIONID). This makes the follow-up requests fail.

Old login page: https://sso.garmin.com/sso/signin

New login page: https://sso.garmin.com/portal/sso/en-US/sign-in

I'm now trying to migrate it to the new login page which does set the correct cookies.

@app4g
Copy link
Author

app4g commented Sep 26, 2023 •
edited
Loading

ah.. yes.. I also saw that, I was previously looking for SESSIONID as well, but like you said, now only SESSION is seen. I wished I had kept a screenshot of the headers when it was working to be able to compare the difference.

with respect to the login page,
when I click the old login page link, and try to login, it will fail

the new login page, It's goes to a "Page Not Found" situation.

when I type connect.garmin.com into the safari, it redirects me to

https://connect.garmin.com/?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F

this works tho

https://sso.garmin.com/portal/sso/en-US/sign-in?clientId=GarminConnect&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern

@app4g
Copy link
Author

app4g commented Sep 27, 2023 •
edited
Loading

I just tried to manually copy the SESSIONID cookie from the webBrowser after logging in and it worked. Essentially, you're right, we need to figure out how to get the SESSIONID cookie again.

I noticed that previously the SESSIONID was made available once we finish authentication (before sending the ticket back to Garmin) and thus we can store that. Now it's not there anymore but it's it one of the ticket URLs (I saw there are 2)

using HTTP Toolkit and browser session..

POST:
https://sso.garmin.com/portal/api/login?clientId=GarminConnect&locale=en-GB&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern
with these data
{
"username": "[email protected]",
"password": "password",
"rememberMe": false,
"captchaToken": "axssfafa"
}

Then GET:
https://connect.garmin.com/modern?ticket=ST-2160819-kqmT7rZ6alD1pRbzV5QE-sso

Then GET:
https://connect.garmin.com/modern/
This is where I start seeing the SESSIOIND cookie

Then GET:
https://sso.garmin.com/sso/login?service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F&webhost=https%3A%2F%2Fconnect.garmin.com&gateway=true&generateExtraServiceTicket=true&generateTwoExtraServiceTickets=true&clientId=GarminConnect

then another Ticket GET request:
https://connect.garmin.com/modern/?ticket=ST-0415002-je4Qp1iQO7ESbIBuWRQj-cas
This has the same SESSIOIND cookie

Then GET:
https://connect.garmin.com/modern/

@stynoo stynoo mentioned this issue Sep 27, 2023
Closed
@app4g app4g mentioned this issue Sep 27, 2023
Closed
@SimonBaars
Copy link
Contributor

Yeah, SESSIONID is all we need. If we add this line after auth:

COOKIE_JAR.set_cookie(http.cookiejar.Cookie(version=0, name='SESSIONID', value='[your_session_id]', port=None, port_specified=False, domain='connect.garmin.com', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False))

... the whole script runs through correctly and activities are exported.

As for obtaining the session id, the new auth flow sends a login POST:

curl 'https://sso.garmin.com/portal/api/login?clientId=GarminConnect&locale=en-US&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern%2F' \
  -H 'content-type: application/json' \
  -H 'cookie: [your_cookies]' \
  --data-raw '{"username":"[uname]","password":"[pass]","rememberMe":false,"captchaToken":""}' \
  --compressed

If we copy the curl from Chrome, all is fine. If we script this, we get a 400 due to the lack of a captcha (even though it's empty in the curl version):

[ERROR] Server couldn't fulfill the request, url https://sso.garmin.com/portal/api/login?clientId=GarminConnect&locale=en-US&service=https%3A%2F%2Fconnect.garmin.com%2Fmodern, code 400, error: HTTP Error 400: Bad Request
Content returned:
%s {"error":"com.garmin.sso.portal.service.ww.exception.InvalidReCaptchaException","errorText":"Recaptcha token is null/empty","errorItems":null,"conversationId":null}

Indeed, once we obtain the ticket from this request, we can use the ticket to obtain a session by calling https://connect.garmin.com/modern/?ticket=[your_ticket]. That request sets the session cookie, as seen in the devtools (or when doing curl verbose):

image

I've tried using Selenium to get the captcha to work, but somehow Garmin detects the use of browser automation. It seems that we're up against a quite robust login system. At least it doesn't have proper replay protection on abovementioned requests.

@SimonBaars
Copy link
Contributor

As for a short-term solution, we could prompt the user for their SESSIONID token instead of uname/pass. Since that token expires, it's not a great solution for automation, what many users seem to use garmin-connect-export for.

I'll continue digging into ways to obtain a valid SESSIONID.

@app4g
Copy link
Author

app4g commented Sep 27, 2023

Funny thing tho, when I use (vanilla) chrome and try a login, it doesn't present me w/ a Captcha and I don't see / have any issues. But when I use HTTPToolKit and it spawns a Chrome session, then the captcha will turn up and I need to do the Captcha.

@SimonBaars
Copy link
Contributor

SimonBaars commented Sep 27, 2023 •
edited
Loading

I think Cloudflare decides how trustworthy your browser is, and part of its cookies is whether captcha validation can be skipped. In case of the chromedriver, it just completely rejects login (the login page loads, but the POST request gives 403).

@SimonBaars
Copy link
Contributor

I created a draft version on my fork of a version that works by prompting for the session id: SimonBaars@6ed2165

It works, but unfortunately this approach appears to be flaky. As in, 1 out of 3 times it will fail due to being unauthenticated, but the behavior is completely inconsistent.

(anyone who would like to try it out: feel free, it should be able to export your activities, but you might need to re-run it if it fails)

@sgowtham
Copy link

sgowtham commented Sep 27, 2023 via email

@SimonBaars
Copy link
Contributor

SimonBaars commented Sep 27, 2023 •
edited
Loading

@sgowtham Doesn't that have the same issue? petergardfjall/garminexport#103

What might be interesting though is the Garth integration: petergardfjall/garminexport#102

@philosowaffle
Copy link

Following along as I am also investigating how to get this resolved for my own project (p2g). While not a solution for me since my project is C#, Garth may be an option for y'all here. The library appears to use the Connect API to support authentication via the more standard OAuth. That being said, skimming through the initial login logic, I think this may hit the same error we are.

@sgowtham
Copy link

sgowtham commented Sep 27, 2023 via email

@SimonBaars
Copy link
Contributor

I did some experimentation with Garth, and it seems like this is an approach that could work! It's able to correctly get oauth tokens and we can use those to get activities.

I'm working on a target implementation now.

@lswiderski lswiderski mentioned this issue Sep 27, 2023
Closed
@SimonBaars SimonBaars mentioned this issue Sep 27, 2023
Merged
@SimonBaars
Copy link
Contributor

SimonBaars commented Sep 27, 2023 •
edited
Loading

We have a candidate solution by migrating to Garth! PR is open ^^

#96

This was referenced Sep 27, 2023
Closed
Merged
@wauwau0977
Copy link

Did get the same issue.

Yet, what I want to say: It's amazing to see how the community driven development works. A big thank you for working on this... 👍

@matin
Copy link

matin commented Sep 27, 2023

Garth maintainer here.

Just read through this thread. Amazing collaboration!

Garth took a lot of effort. Happy to hear it resolved the auth issues!

A recommendation for future dev: saving and reusing the OAuth tokens (instructions in the README). The OAuth1 token Garth obtains during the login is valid for a year. This is particularly important for people like me who use MFA.

Let me know how I can be helpful for using Garth or migrating it to other languages.

@dylix
Copy link

dylix commented Sep 27, 2023

One thing I noticed from @SimonBaars https://github.com/SimonBaars/garmin-connect-export/tree/use-garth-oauth

(which I'm using now for my script :P)

URL_GC_ORIGINAL_ACTIVITY = 'http://connect.garmin.com/download-service/files/activity/'
was unchanged, it still had the 'proxy/' in the URL.. removing it, and everything is working again!

What an amazing community thx so much for all the fixes!

@philosowaffle
Copy link

@matin Hi, thank you for the awesome work on Garth! I'm looking at porting this over to C# right now and came across this line. If I understand correctly, if Garmin revokes this key/secret pair then we'll all be broken again?

@matin
Copy link

matin commented Sep 27, 2023

@matin Hi, thank you for the awesome work on Garth! I'm looking at porting this over to C# right now and came across this line. If I understand correctly, if Garmin revokes this key/secret pair then we'll all be broken again?

The consumer key and secret are in S3 (instead of the code) to make sure they can be updated at any given time to ensure they're active.

@philosowaffle
Copy link

Makes sense. given Garmin's history of gatekeeping their API's to only "approved business developers", I'm concerned once they see how these creds are being used, they will revoke and not issue new ones.

Not saying we currently have a better option. Just documenting the risk.

@matin
Copy link

matin commented Sep 27, 2023

To be clear, these are not creds issued to me for business purposes. They're from the Garmin Connect app. They would only be updated if they're also updated in the app. In which case, I'll update with the new ones.

@SimonBaars
Copy link
Contributor

@dylix Thanks for checking it out, I fixed the issue you mentioned!

@matin
Copy link

matin commented Sep 28, 2023 •
edited
Loading

@SimonBaars I've said it before, but it bears repeating: great work on migrating other projects to Garth so quickly!

Given the broader use of garth.sso, I just released 0.4.30 (and now 0.4.31) to make the authentication more robust. The the OAuth1Session now uses the same retry and backoff logic as the session used to make requests.

Custom timeout and retry logic can be set with garth.configure().

Here are the defaults:

class Client:
    ...
    timeout: int = 10
    retries: int = 3
    status_forcelist: Tuple[int, ...] = (408, 429, 500, 502, 503, 504)
    backoff_factor: float = 0.5
    ....

This was referenced Sep 28, 2023
Closed
Closed
@teibaz teibaz mentioned this issue Sep 29, 2023
Open
@jlouder jlouder mentioned this issue Sep 29, 2023
Closed
@telemaxx
Copy link
Contributor

@matin I have also the problem with our garminexport script.

so yesterday, I tried to install garth on debian Bookworm, but I have had no luck so far.

error was something about python environment. I will investigate further and report.

is garth working with python3?

@telemaxx
Copy link
Contributor

i did this:

sudo apt install python3-pip

pip3 install garth --user

and i got a long error message:

$ pip3 install garth --user
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.

If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.

See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

@matin
Copy link

matin commented Sep 30, 2023

@matin I have also the problem with our garminexport script.

so yesterday, I tried to install garth on debian Bookworm, but I have had no luck so far.

error was something about python environment. I will investigate further and report.

is garth working with python3?

Garth is available on Python 3.8 and higher. Try python -m pip install garth instead of pip3 install garth.

If you still run into issues, create a GH issue for Garth, and I can help you there.

@matin matin mentioned this issue Sep 30, 2023
Open
@Haryt
Copy link

Haryt commented Oct 1, 2023

Hi, have the same Problem since 2 or 3 days.
I installed garth (on windows 10, python 3.11, garmin export is the version from moderation) without success.

This is the errormessage:

ile "Z:\Sport\Script\gcexport.py", line 365, in
ACTIVITY_LIST = http_req(URL_GC_LIST + urllib.parse.urlencode(SEARCH_PARAMS))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Z:\Sport\Script\gcexport.py", line 165, in http_req
response = OPENER.open(request, data=post)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\urllib\request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\urllib\request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\urllib\request.py", line 563, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\urllib\request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\urllib\request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

@telemaxx
Copy link
Contributor

telemaxx commented Oct 1, 2023 •
edited
Loading

@matin I have also the problem with our garminexport script.
so yesterday, I tried to install garth on debian Bookworm, but I have had no luck so far.
error was something about python environment. I will investigate further and report.
is garth working with python3?

Garth is available on Python 3.8 and higher. Try python -m pip install garth instead of pip3 install garth.

If you still run into issues, create a GH issue for Garth, and I can help you there.

Thank you for you're answer @matin.

i found a solution/workarround on Stackoverflow

as mentioned there i renamed one file(i dont want to delete it)
sudo mv /usr/lib/python3.11/EXTERNALLY-MANAGED /usr/lib/python3.11/EXTERNALLY-MANAGED.OFF

than i was able to install garth:
python3 -m pip install garth

next i have to implement garth in my garmin-connect-export fork
that fork i a bit old, but with some extras which i need.

i created an issue at youre garth project, maybe you can give an installation hint in the docu.

@telemaxx
Copy link
Contributor

telemaxx commented Oct 1, 2023

Hi, have the same Problem since 2 or 3 days. I installed garth (on windows 10, python 3.11, garmin export is the version from moderation) without success.

did you mean the branch moderation?

Actually the PR with garth is not merged.
so we have to wait a bit...

@Haryt
Copy link

Haryt commented Oct 2, 2023

Yes, i men the moderation branch, It seem that this branch is dead because since 1 year there is no change and also no answer to my question.... So I have to change to pe-st in the future.

@moderation
Copy link
Contributor

Yes, i men the moderation branch, It seem that this branch is dead because since 1 year there is no change and also no answer to my question.... So I have to change to pe-st in the future.

@Haryt my moderation branch isn't dead. I've had a recent cycling injury and am still getting back up to speed. I will incorporate the changes from this branch into mine shortly.

@Haryt
Copy link

Haryt commented Oct 3, 2023

@moderation
Good to hear that the branch is living. Whish you all the best for recovery!

@app4g
Copy link
Author

app4g commented Oct 3, 2023

FYI, the folks here petergardfjall/garminexport#103 (comment) has managed to get it sorted out w/o using garth.

@telemaxx telemaxx mentioned this issue Oct 6, 2023
Closed
@pe-st
Copy link
Owner

pe-st commented Oct 8, 2023

Should be fixed with PR #96 (merged into v4.2.0)

@pe-st pe-st closed this as completed Oct 8, 2023
@moderation
Copy link
Contributor

@moderation Good to hear that the branch is living. Whish you all the best for recovery!

@Haryt I've finally gotten around to updating my fork. I borrowed heavily from @pe-st work with Garth. I just tested on my last 3 rides and it seems to be working.

https://github.com/moderation/garmin-connect-export/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests