PyiCloudAPIResponseException('Service Temporarily Unavailable (503)')

[Echoooo0827]

The problem

I ran the program and got a 'Service Temporarily Unavailable (503)' exception

Environment

• pyiCloud release with the issue (pip show pyicloud): Version: 1.0.0
• Last working pyiCloud release (if known): Version: 1.0.0
• Service causing this issue: FindMyPhone
• Python version (python -V): python-3.10.8
• Operating environment (project deps/Docker/Windows/etc.): windows

Traceback/Error logs

Traceback (most recent call last):
  File "C:\Work\Env\python\Lib\site-packages\pyicloud\base.py", line 325, in authenticate
    self.session.post(
  File "C:\Work\Env\python\Lib\site-packages\requests\sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
  File "C:\Work\Env\python\Lib\site-packages\pyicloud\base.py", line 133, in request
    self._raise_error(response.status_code, response.reason)
  File "C:\Work\Env\python\Lib\site-packages\pyicloud\base.py", line 189, in _raise_error
    raise api_error
pyicloud.exceptions.PyiCloudAPIResponseException: Service Temporarily Unavailable (503)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Work\pythonProjects\demo\pythonProject\find-my-phone-demo.py", line 3, in <module>
    api = PyiCloudService('****', '*******')
  File "C:\Work\Env\python\Lib\site-packages\pyicloud\base.py", line 271, in __init__
    self.authenticate()
  File "C:\Work\Env\python\Lib\site-packages\pyicloud\base.py", line 333, in authenticate
    raise PyiCloudFailedLoginException(msg, error) from error
pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid email/password combination.', PyiCloudAPIResponseException('Service Temporarily Unavailable (503)'))

Checklist

• I've looked informations into the README.
• I've looked informations into the pyiCloud's code.
• I've looked informations in Google.

Additional information

from pyicloud import PyiCloudService

api = PyiCloudService('***', '****')

if __name__ == '__main__':
    print(api)

[mariuslacatus]

Any traction on this? I just tried to pull some of my photos from iCloud and I'm getting this error.

[bungasnail]

I am in the same situation.
Is this a problem with the execution environment?
Is there any way to improve it?

[tomballgithub]

Same here

[matzekl]

Same issue here only since a few days

[pop2pop3]

Same issue here

[tomballgithub]

This app identified a change needed but the apple-tools repository with the fix is not public.
https://status.expo.dev/incidents/t3cwh0l0x5qd?u=9bkhv9wfr16k

[pop2pop3]

I think it's indeed the authentication problem where the AUTH_ENDPOINT URL in pyicloud base.py file seems no longer valid.

[lupojpoodle]

Authentication is now SRP6a. Old UID/pwd method gets the 503.

[Aulig]

This app identified a change needed but the apple-tools repository with the fix is not public. https://status.expo.dev/incidents/t3cwh0l0x5qd?u=9bkhv9wfr16k

You can check out Fastlane who have a fix now: fastlane/fastlane@faee8dc

It's in ruby, but at least public. But I haven't been able to convert it to python yet either.

[iowk]

I have managed to fix the authentication in another project that forks pyicloud. See commit icloud-photos-downloader/icloud_photos_downloader@4bcb2ac

Just want to share it as a reference if anyone else is working on this issue.

[pop2pop3]

I have managed to fix the authentication in another project that forks pyicloud. See commit icloud-photos-downloader/icloud_photos_downloader@4bcb2ac

Just want to share it as a reference if anyone else is working on this issue.

Can you rewrite the pyicloud base.py with what you've done in icloudpd base.py?

Huge thanks for the work!

[swb1701]

base.txt

This worked for me as a replacement for base.py in pyicloud. Sorry, not set up to do a pull request or the like. Maybe someone could make a new pyicloud release.

Sorry, delete the headers2 = definition. Forgot to clean that up.

[DartVador]

base.txt

This worked for me as a replacement for base.py in pyicloud. Sorry, not set up to do a pull request or the like. Maybe someone could make a new pyicloud release.

Sorry, delete the headers2 = definition. Forgot to clean that up.

it works for me !!
Thanks

[pop2pop3]

I have created a PR if somehow approved

Many thanks to @iowk who have done great efforts to fix the authentication method.

See commit: 41be37d

[DSPhotography]

base.txt
This worked for me as a replacement for base.py in pyicloud. Sorry, not set up to do a pull request or the like. Maybe someone could make a new pyicloud release.
Sorry, delete the headers2 = definition. Forgot to clean that up.

it works for me !! Thanks

Hey @DartVador, I'm still new-ish to all of this. Can you give a general idea of what you did to use the base.txt that he created? I'm assuming from the looks of the file that I'm just copying the file/contents and replacing the code in my current setup... I'm just not fully understanding where that code is that needs to be replaced or updated by this new file. I'm heading to google now, so please forgive my order of asking for help vs. searching on my own. I'm multitasking this with also playing dad to a 3yr old and a 3month old.

[DartVador]

base.txt
This worked for me as a replacement for base.py in pyicloud. Sorry, not set up to do a pull request or the like. Maybe someone could make a new pyicloud release.
Sorry, delete the headers2 = definition. Forgot to clean that up.

it works for me !! Thanks

Hey @DartVador, I'm still new-ish to all of this. Can you give a general idea of what you did to use the base.txt that he created? I'm assuming from the looks of the file that I'm just copying the file/contents and replacing the code in my current setup... I'm just not fully understanding where that code is that needs to be replaced or updated by this new file. I'm heading to google now, so please forgive my order of asking for help vs. searching on my own. I'm multitasking this with also playing dad to a 3yr old and a 3month old.

the base.txt file must be renamed to base.py.
then you have to put it in the directory where the lib is installed.
For me it was in ~/.local/lib/python3.9/site-packages/pyicloud/
save the original file before copying!

[steal25]

/.local/lib/python3.9/site-packages

Thank you for this info! But im VERY new to all this and after searching for about an hour or so i havent found any help on how to actually move the file into the directory, or actually anything that i feel comfortable doing without thinking ill mess up the whole thing....hahaha!. if you could guide me on how to do that, that would be great! im running home assistant os on a mini beelink pc if that helps. thank you.

[tomballgithub]

I took my same setup as always, installed the srp module, replaced base.py, but no matter what I do I get this:
pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))

I even reinstalled a new python instance but get the same thing.

Any ideas?

[tomballgithub]

I took my same setup as always, installed the srp module, replaced base.py, but no matter what I do I get this: pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))

I even reinstalled a new python instance but get the same thing.

Any ideas?

It only gives that error for my account without 2FA
It works as expected on a 2nd account with 2FA.
I presume this is now expected behavior

[fabricecatherine03]

Hi,
@swb1701 : many thanks for the adaptation of base.py. Very helpful.
However, I am facing the same issue as @tomballgithub. I get the same error message, despite my account is 2FA:
pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', ?
I have no App-Specific Passwords.
Anyone has an idea how to help me to solve this issue ?
Thanks

[Gdocal]

same issue Missing apple_id field for account with 2FA

[fabricecatherine03]

Even if my account is 2FA, I have tried to modify the password. It does not help as I continue to get the error "missing apple_id field"

[BS-98]

Hi,

I have 2FA account. With new base.py file from @pop2pop3 (commit 41be37d) authorization worked, but with "normal" user's password. Before I tried with app-specific password and it did not work. I got pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))

[pop2pop3]

Hi,

I have 2FA account. With new base.py file from @pop2pop3 (commit 41be37d) authorization worked, but with "normal" user's password. Before I tried with app-specific password and it did not work. I got pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field'))

That also happened to me after having the base.py code modified, and tried to run the custom code several times to test if my find my iphone works (I also have 2FA account) and it finally worked out, then I add an argument cookie_directory to save my session token, and recall that file to run the custom code for another session, and fortunately the code still works until now, and everytime I use the API, it won't call 2FA anymore. Note that I didn't even change my iCloud account password, settings, whatsoever.

[Gdocal]

After spending a whole day on this bug, I found solutions for two issues:

1. I have two iCloud accounts. Login works fine for one, but for another the Apple server returns:

{
  "serviceErrors" : [ {
    "code" : "-20101",
    "message" : "Enter the email or phone number and password for your Apple Account.",
    "suppressDismissal" : false
  } ]
}

The pyicloud library raises pyicloud.exceptions.PyiCloudFailedLoginException: ('Invalid authentication token.', PyiCloudAPIResponseException('Missing apple_id field')) for this response.

2. For the account which I can login to, I was not able to make my session trusted ("hsaTrustedBrowser":false). I wasn't sure if others were experiencing the same trusted session issue as I use own fix for initial login issue.

Today I started implementing iCloud login from scratch, trying to mimic real browser requests as closely as possible, with all headers etc. I thought library might be missing some headers under specific circumstances.

So I tried to make requests as close as possible to the browser so I could compare requests in Fiddler and spot any differences. This really helped me fix the session trusting issue. (For now I haven't identified what exactly needs to be fixed in pyicloud to make it work like my code.)

But this new code didn't fix login for my other account at all. I was able to login to it through a browser so it's definitely not an issue on the server side, like they didn't update something on their side.
The issue is in the SRP implementation, but there's no easy way to test as data sent to server and received is always different. I decided to look at the JS implementation of SRP and find differences.

The SRP code is located at https://appleid.cdn-apple.com/appleauth/static/jsj/445765738/webSRPClientWorker.js
After debugging and comparing how SRP works between Python and JS by using the same private and public keys from JS, I got identical M1 (Client Proof) and M2 (Server Proof), but for the other account I get these values different.

So the issue is in the Python SRP implementation. After digging deeper, I found that the issue is in the salt parameter. It seems the salt is always the same for the same account, and the issue occurs when the salt starts with 0 bytes in base64 (it starts with "AA"). The SRP library in Python converts salt bytes to a long number and when it calculates x (private key multiplier) it converts from number to bytes array again, but we lose the leading zero bytes and use completely different bytes for the salt.

Here is the fix for the SRP library: https://gist.github.com/Gdocal/45c24f542ebd32a2256217bfd856d972 and original one is here https://github.com/cocagne/pysrp/blob/master/srp/_pysrp.py
also i make SRP lib to work with 256 bytes private key instead of 32 bytes. Not sure it is important.

[magicus]

@Gdocal Great find! Have you filed a bug upstream to the SRP library?

[Gdocal]

@Gdocal Great find! Have you filed a bug upstream to the SRP library?

I am not sure about specification is salt in srp protocol allowed to start from zero bytes or it is just a decision/buf from apple

[AndreyNikiforov]

@Gdocal Great find! Have you filed a bug upstream to the SRP library?

I am not sure about specification is salt in srp protocol allowed to start from zero bytes or it is just a decision/buf from apple

Example in RFC has salt as bytes, not as string, so I assume omission of leading bytes is incorrect. Spec does not explicitly tell that though.

IMO posting question/concern to upstream srp project is the way to go.

[PaulCavill]

@balloob

[magicus]

I took @Gdocal's patch, cleaned it up a bit, and posted it as a PR to the pysrp project: cocagne/pysrp#61

@Gdocal I hope you don't mind; I tried my best to make sure you are properly credited. I just thought it was important to get this fixed upstream ASAP, since other projects (like https://github.com/gcobb321/icloud3_v3) also depend on a properly working pysrp.

[plk]

Has anyone tested the srp patch? After patching base.py I get the "missing apple_id" error but patching the srp lib as above doesn't seem to fix this?

[tomballgithub]

I implemented the patch last night and it didn't break my setup that was already working.

…

On Fri, Nov 1, 2024 at 12:40 PM plk ***@***.***> wrote: Has anyone tested the srp patch? After patching base.py I get the "missing apple_id" error but patching the srp lib as above doesn't seem to fix this? — Reply to this email directly, view it on GitHub <#456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADVRWS5BWOMMGQMI73QPAJDZ6O4KBAVCNFSM6AAAAABMH255DCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJSGMYDGNBZGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[plk]

Hmm, I patched both base.py and _pysrp.py and I still get the missing apple_id error. No app-specific password, 2FA enabled.

[tomballgithub]

Hmm, I patched both base.py and _pysrp.py and I still get the missing apple_id error. No app-specific password, 2FA enabled.

Did you change your password recently (regardless of 2FA)? That fixed mine

[plk]

Just did it and now get this in debug log:

DEBUG:pyicloud.base.http:{'serviceErrors': [{'code': '-20101', 'message': 'Enter the email or phone number and password for your Apple\xa0Account.', 'suppressDismissal': False}]}

I think the password reset is because you get locked out when testing this over and over. I could login via web GUI etc. but the python debug log clearly said the account was locked out which is why resetting the password changed the error. Now I'm stuck at the above error, despite having patched srp.

[tomballgithub]

Which base.py are you using? I saw two different ones in two different pull requests.

…

On Fri, Nov 1, 2024 at 2:56 PM plk ***@***.***> wrote: Just did it and now get this in debug log: DEBUG:pyicloud.base.http:{'serviceErrors': [{'code': '-20101', 'message': 'Enter the email or phone number and password for your Apple\xa0Account.', 'suppressDismissal': False}]} — Reply to this email directly, view it on GitHub <#456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADVRWS46EPK4S66PVCBAW53Z6PMGDAVCNFSM6AAAAABMH255DCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJSGUYDEMBXG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[cocagne]

I've updated pysrp to version 1.0.22 which includes the patch from @Gdocal & @magicus. Please let me know if errors are found. i don't have a good way to test the code.

[Gdocal]

Hmm, I patched both base.py and _pysrp.py and I still get the missing apple_id error. No app-specific password, 2FA enabled.

make sure to clear all cache, and run code again. I had an issue with this when patched a lib.

[plk]

Latest srp patch, tried both patches for base.py, cleared all caches. Same issue as above - always "missing apple_id" field caused by the "Enter the email ... " prompt in the session building stage.

[PaulCavill]

I had the same issue where the Error "Missing apple_id" in Home Assistant and I had to define the arguments.

self.api = PyiCloudService( apple_id=self._username, password=self._password, cookie_directory=self._icloud_dir.path, with_family=self._with_family, )

[plk]

Hmm, doesn't work for me - I'm calling this anyway:

api = PyiCloudService('<username>, <password>)

If I add the apple_id to the data in the _authenticate_with_token call it complains it's missing the password, if I add that there too, it then gives a 421 error "Authentication required for Account".

[Gdocal]

Latest srp patch, tried both patches for base.py, cleared all caches. Same issue as above - always "missing apple_id" field caused by the "Enter the email ... " prompt in the session building stage.

You can contact me in telegram @Gdocal26 so we can investigate your issue .

[gcobb321]

@cocagne

I've updated pysrp to version 1.0.22 which includes the patch from @Gdocal & @magicus. Please let me know if errors are found. i don't have a good way to test the code.

I have included the _pysrp patched code with the latest iCloud3 beta. I think it is v1.0.22 but it would be helpful if that was included in the comments at the beginning of the code. It seems to be working but I have one user who is getting a 401 from the …/signing/complete call to Apple. The username/password is valid as confirmed by a call to … setup/authenticate/username@… I think this would indicate a hash key calculation error. I just sent him new code to catch that condition and suggested he try another account or change passwords.

Any thoughts on this? Is there any data that might identify the problem? What is the github repository I can fork with the latest code?

[cocagne]

The easiest way to verify what version of pysrp you're using is probably to just do a "pip freeze" from the command line. That'll show you the exact version of every Python package you have installed. The code can be found here: https://github.com/cocagne/pysrp. I'm not familiar with the icloud application so I'm afraid I won't be of much help in that regard. Good luck!

…

On Sun, Nov 3, 2024 at 12:59 PM Gary Cobb ***@***.***> wrote: @cocagne <https://github.com/cocagne> I've updated pysrp to version 1.0.22 which includes the patch from @Gdocal <https://github.com/Gdocal> & @magicus <https://github.com/magicus>. Please let me know if errors are found. i don't have a good way to test the code. I have included the _pysrp patched code with the latest iCloud3 beta. I think it is v1.0.22 but it would be helpful if that was included in the comments at the beginning of the code. It seems to be working but I have one user who is getting a 401 from the …/signing/complete call to Apple. The username/password is valid as confirmed by a call to … setup/authenticate/username@… I think this would indicate a hash key calculation error. I just sent him new code to catch that condition and suggested he try another account or change passwords. Any thoughts on this? Is there any data that might identify the problem? What is the github repository I can fork with the latest code? — Reply to this email directly, view it on GitHub <#456 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANMW7FYKVG36WCZIYEICNLZ6ZP7ZAVCNFSM6AAAAABMH255DCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINJTGU2DCMRUGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gcobb321]

@cocagne
Thanks for the info. I have forked the pysrp repository and am watching it for any notifications and activity.

My program, iCloud3 (docs here), tracks the location and other data of iPhones, iPads, Apple Watches and other iDevices and runs as a component of Home Assistant, a program that manages home automation functions. It is responsible for the Python library running on its OS and I do not want to depend on it for pysrp support. I have copied the raw code into a file that I include with my package.

[PaulCavill]

@Gdocal any ideas when we will see a new release for this.