rework authentication and API structure after VW has switched off CarNet #216

oliverrahner · 2023-11-28T14:47:20Z

No description provided.

volkswagencarnet/vw_connection.py

robinostlund · 2023-11-28T15:37:02Z

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

den4uk · 2023-11-28T16:55:22Z

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

Could you do a pre-release, so it would be possible to install it via HACS, and tested by wider audiences?

oliverrahner · 2023-11-28T16:55:29Z

Thanks @robinostlund
I currently have access to a diesel, a hybrid and an electric car, and I’m also trying to cater for some things like missing glass roof and the like.
I tend to go for one release once I have „all“ entities readable, and another one for actually controlling settings.

oliverrahner · 2023-11-28T16:57:37Z

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

Could you do a pre-release, so it would be possible to install it via HACS, and tested by wider audiences?

Unfortunately, it’s not the integration I’m reworking, but the underlying library.
I don’t have a creative idea right now how we could make these „alphas“ available for preview… do you, @robinostlund?

den4uk · 2023-11-28T16:59:27Z

@oliverrahner i let you decide when you feel this is ready for release, i believe it may be better to release it once when it is sort of working as i guess people will find missing things as there are different values depending on which type of car you have 😄

Could you do a pre-release, so it would be possible to install it via HACS, and tested by wider audiences?

Unfortunately, it’s not the integration I’m reworking, but the underlying library. I don’t have a creative idea right now how we could make these „alphas“ available for preview… do you, @robinostlund?

The owner (@robinostlund) could take a tag of your branch, and create a pre-release. I would test it on my ID.3 tonight.

TheBlackBGD · 2023-11-28T17:04:27Z

I would test it also.

LucTs · 2023-11-28T17:20:42Z

Mee too, I do have a 2022 Touran.

robinostlund · 2023-11-28T17:53:35Z

Is it ok if i do it tomorrow morning? Dont have any computer infront of me now 😊

@oliverrahner i think you should have access to do this also now when you are a collaborator 😊

TheBlackBGD · 2023-11-28T18:09:26Z

Let us know when we can update from hacs ill do it roght away, for me is also ok for tomorrow as i see @oliverrahner used all my apicalls and the vw application is frozen 😂😂😂
No worries :)

virtualdj · 2023-11-28T18:19:55Z

I don’t have a creative idea right now how we could make these „alphas“ available for preview…

Actually if one runs the Python code in the example with the library saved as a subfolder ("download as ZIP") and with the correct login information, we/you could get information about the sensors that are working or not.

It is not the easy of use of Home Assistant sensors, but for debugging a lot of configurations it could be a viable solution.

volkswagencarnet/vw_vehicle.py

oliverrahner · 2023-11-28T18:35:37Z

Let us know when we can update from hacs ill do it roght away, for me is also ok for tomorrow as i see @oliverrahner used all my apicalls and the vw application is frozen 😂😂😂 No worries :)

That is very strange... I tested only very little with your car and mostly with mine and the one from @michiii1337, and ours still work... But I can confirm that I also get HTTP 429s from the VW API for your credentials.

We need to monitor that, because I have set up all our 3 cars in my Dev HA environment, so it might be some case that produces more API calls for electric cars than for others.

TheBlackBGD · 2023-11-28T18:40:22Z

Let us know when we can update from hacs ill do it roght away, for me is also ok for tomorrow as i see @oliverrahner used all my apicalls and the vw application is frozen 😂😂😂 No worries :)

That is very strange... I tested only very little with your car and mostly with mine and the one from @michiii1337, and ours still work... But I can confirm that I also get HTTP 429s from the VW API for your credentials.

We need to monitor that, because I have set up all our 3 cars in my Dev HA environment, so it might be some case that produces more API calls for electric cars than for others.

@oliverrahner it is possible i dont know i use the app also almoast every time i go somewhere to heat the car, and i have also the other integration installed. But to have some ideeas i will delete the other integration and see from tomorrow what is hapening, ok?

oliverrahner · 2023-11-28T18:41:48Z

@oliverrahner it is possible i dont know i use the app also almoast every time i go somewhere to heat the car, and i have also the other integration installed. But to have some ideeas i will delete the other integration and see from tomorrow what is hapening, ok?

Sounds good! I also removed it now from my Dev instances, so we can investigate better.

TheBlackBGD · 2023-11-28T18:58:24Z

@oliverrahner it is possible i dont know i use the app also almoast every time i go somewhere to heat the car, and i have also the other integration installed. But to have some ideeas i will delete the other integration and see from tomorrow what is hapening, ok?

Sounds good! I also removed it now from my Dev instances, so we can investigate better.

@oliverrahner I have deleted the other integration. As it happened last time after midnight it worked again, so from now it will be my mobile app and you for the api calls. If i can help with something let me know!

volkswagencarnet/vw_connection.py

volkswagencarnet/vw_vehicle.py

den4uk · 2023-11-28T19:30:05Z

volkswagencarnet/vw_vehicle.py

+        data = await self._connection.getParkingPosition(self.vin)
+        if data:
+            self._states.update(data)


Suggested change

data = await self._connection.getParkingPosition(self.vin)

if data:

self._states.update(data)

if data := await self._connection.getParkingPosition(self.vin):

self._states.update(data)

Just a perfect use for a walrus operator 😄

Same argument again :) I tried to copy the original style as much as possible, and that's how it was done before.
My perspective on this is: rather have uniform code than optimizing every little detail once you stumble over it.

Sorry, resolved the wrong convo

stickpin · 2023-12-05T17:23:22Z

Interestingly, I am calling parkingposition all day long every 5 minutes, and still have not reached the rate limit.

@stickpin Ok now this is really interesting... have you tried spamming the API with the script I posted above? Because that API was the only one where I was able to artificially hit the rate limit.

No, I am using the integration which is running 4 API's every 5 minutes + Login flow or Refresh token flow every hour.

oliverrahner · 2023-12-05T17:25:24Z

Gets confusing now :) I'll repost my comments in the PR, where they belong and remove them here.

Regarding the rate limit: Could you try to spam the parking position API? I'll update the script in a second. Because I would really like to know if you can also reach the limit and at how many requests.

oliverrahner · 2023-12-05T17:29:07Z

I updated the Gist:
https://gist.github.com/oliverrahner/4c9b616b1f934ac7df9ec423b78723dc

stickpin · 2023-12-05T17:41:24Z

I updated the Gist: https://gist.github.com/oliverrahner/4c9b616b1f934ac7df9ec423b78723dc

The problem with this script is that is hammering the server every second. That might be the reason why you are getting rate-limited. I am not saying that there is no rate limit, I am just saying that there might be a combination of both amount of API requests and how frequently you do it. At least this is what I would do on WAF. :)

oliverrahner · 2023-12-05T17:51:22Z

Now the change I intended is there, 6d12a41... Difficult decision if we should leave it in or not. It doesn't seem to make a difference for you, but I'm also look at all the other people complaining about being rate limited 🤔

I agree that the actual rate and not only the pure number of requests should make a difference, but then it doesn't explain why the API doesn't bother at all about the other endpoints being hammered in the same way, but the do limit the parking position one...

oliverrahner · 2023-12-05T17:52:47Z

Right now I am at 820 requests to parkingposition in a row, with some occasional 403s in between, but that happens on all services and recovers after a short time... so now I can't even reproduce hitting the limit with that 1s interval 😬

stickpin · 2023-12-05T17:56:59Z

Right now I am at 820 requests to parkingposition in a row, with some occasional 403s in between, but that happens on all services and recovers after a short time... so now I can't even reproduce hitting the limit with that 1s interval 😬

it might also be that VW moved to Prod and is still fine-tuning the configs.
I guess I will move my setup from dev to prod in will run with the get_parkingposition() with default 5-minute intervals, let's see if I will hit the rate limit.

virtualdj · 2023-12-06T12:27:25Z

Difficult decision if we should leave it in or not. It doesn't seem to make a difference for you, but I'm also look at all the other people complaining about being rate limited 🤔

@oliverrahner
Isn't it possible to pass a flag to the library to instruct it whether it should poll vehicle_moving or not?

If a user has the vehicle_moving entity added and enabled in HA, then poll. Otherwise, use the other algorithm based on the last trip to produce less API calls.

In this way both kind of users will be happy...

stickpin · 2023-12-08T10:31:03Z

@oliverrahner I don't know if you paid attention but currently if VW servers go down, all the sensors are still available, and it gives you the wrong impression that everything is fine. We can introduce the connection status sensor that will show the service status. I have such a sensor for my Bosch/Siemens home appliances. OR we can drop all the sensors in an unavailable state which will indicate the problem. OR maybe even both. I think I like the last option more, connection status sensor + all the sensors in an unavailable state.

I will work on the implementation of both. Let me know if you see it differently.

oliverrahner · 2023-12-08T11:49:56Z

I liked to proposal.
What do you say, you start working on that status stuff and in the meantime I can look at restoring the control functions?

oliverrahner · 2023-12-08T11:50:53Z

If a user has the vehicle_moving entity added and enabled in HA, then poll. Otherwise, use the other algorithm based on the last trip to produce less API calls.

In this way both kind of users will be happy...

@virtualdj Just noticed I didn't answer. Right now, we hope that @stickpin's fix really solves the problem in total, so that the users don't need to make that choice.

stickpin · 2023-12-08T11:52:37Z

I liked to proposal. What do you say, you start working on that status stuff and in the meantime I can look at restoring the control functions?

yes, sounds good. :)

stickpin · 2023-12-08T14:41:31Z

@oliverrahner well...
There is an extra logic to persist the previous state in the frontend integration:

    def async_write_ha_state(self) -> None:
        """Write state to HA, but only if needed."""
        backend_refresh_time = self.instrument.last_refresh
        # Get the previous state from the state machine if found
        prev: Optional[State] = self.hass.states.get(self.entity_id)

        # This is not the best place to handle this, but... :shrug:..
        if self.attribute == "requests_remaining" and self.state in [-1, STATE_UNAVAILABLE, STATE_UNKNOWN]:
            restored = prev or self.restored_state
            if restored is not None:
                try:
                    value = int(restored.state)
                    _LOGGER.debug(f"Restoring requests remaining to '{restored.state}'")
                    self.vehicle.requests_remaining = value
                except ValueError:
                    pass
            else:
                _LOGGER.debug(f"Not changing requests remaining to '{self.state}'")
                return

        # need to persist state if:
        # - there is no previous state
        # - there is no information about when the last backend update was done
        # - the state has changed (need to do string conversion here, as "new" state might be numeric,
        #   but the stored one is a string... sigh)
        if (
            prev is None
            or str(prev.attributes.get("last_updated", None)) != str(backend_refresh_time)
            or str(self.state or STATE_UNKNOWN) != str(prev.state)
        ):
            super().async_write_ha_state()
        else:
            _LOGGER.debug(f"{self.name}: state not changed ('{prev.state}' == '{self.state}'), skipping update.")

Then I guess I will keep it this way and introduce only a connection status sensor for the time being.

@robinostlund what was the initial idea to keep the previous state of the sensors if VW servers are down?

stickpin · 2023-12-08T15:44:39Z

@oliverrahner the funny part is that every VW endpoint is acting differently.

Now for example:

authorize - HTTP 200
token - HTTP 200
vehicles - HTTP 200
parkingposition - HTTP 502
capabilities - HTTP 504

Which connection status we should choose? :))

Or maybe create some string-based sensor and have 3 statuses? "Up", "Down", "Warning" with attributes that will show each endpoint status?

stickpin · 2023-12-08T16:13:54Z

Attributes are a bad idea. But to create a binary_sensor for each endpoint is also not very elegant.
But this solution might show a real status on the platform.

I guess I will start moving in this direction.

oliverrahner · 2023-12-08T16:27:02Z

That's also why I had a hard time reacting to the request for "last update" time that some people mentioned... there is not a specific time for everything, but rather a specific time per endpoint, sometimes even below that... I don't have a clear answer to this yet

stickpin · 2023-12-08T16:45:12Z

@oliverrahner wow. now I managed to get a response from selectivestatus like this:
HTTP Status 207

...
    "oilLevel": {
        "oilLevelStatus": {
            "error": {
                "message": "Bad Gateway",
                "errorTimeStamp": "2023-12-08T16:32:42Z",
                "info": "Something went wrong. Please try to re-login. If the problem persists, please contact our support.",
                "code": 4004,
                "group": 3,
                "retry": true
            }
        }
    },
    "measurements": {
        "rangeStatus": {
            "value": {
                "carCapturedTimestamp": "2023-12-08T14:58:25Z",
                "gasolineRange": 520,
                "totalRange_km": 520
            }
        },
        "odometerStatus": {
            "value": {
                "carCapturedTimestamp": "2023-12-08T14:58:25Z",
                "odometer": 33922
            }
        },
...

hmmm... one more use case to consider... :(

stickpin · 2023-12-08T18:28:35Z

@oliverrahner what do you say?

The current logic is:

        if response_code == 200 or response_code == 204:
            status = "Up"
        elif response_code == 207:
            status = "Warning"
        elif response_code == 429:
            status = "Rate limited"
        else:
            status = "Down"

stickpin · 2023-12-08T19:10:50Z

@oliverrahner please check: #228

I think we can merge it into the next stable branch as well.
So in case people complain, they will be able to see the status of the APIs by themselves. :)

stickpin · 2023-12-08T20:52:15Z

@oliverrahner wow. now I managed to get a response from selectivestatus like this: HTTP Status 207

...
    "oilLevel": {
        "oilLevelStatus": {
            "error": {
                "message": "Bad Gateway",
                "errorTimeStamp": "2023-12-08T16:32:42Z",
                "info": "Something went wrong. Please try to re-login. If the problem persists, please contact our support.",
                "code": 4004,
                "group": 3,
                "retry": true
            }
        }
    },
    "measurements": {
        "rangeStatus": {
            "value": {
                "carCapturedTimestamp": "2023-12-08T14:58:25Z",
                "gasolineRange": 520,
                "totalRange_km": 520
            }
        },
        "odometerStatus": {
            "value": {
                "carCapturedTimestamp": "2023-12-08T14:58:25Z",
                "odometer": 33922
            }
        },
...

hmmm... one more use case to consider... :(

@oliverrahner Ignore it... this API always returns 207. I will update API status codes.

robinostlund · 2023-12-18T09:24:57Z

@oliverrahner should we close this PR?

oliverrahner · 2023-12-18T15:21:20Z

All relevant work of this PR has been merged as part of #226

oliverrahner added 2 commits November 28, 2023 15:45

reworked authentication and many properties

fa720bd

revert well-meant renaming of parameter

f2e5f08

oliverrahner mentioned this pull request Nov 28, 2023

[BUG] Login for VW account failed! robinostlund/homeassistant-volkswagencarnet#507

Closed

robinostlund added the bug Something isn't working label Nov 28, 2023

adding check on secondary drive

aa1fc90

den4uk reviewed Nov 28, 2023

View reviewed changes

volkswagencarnet/vw_connection.py Outdated Show resolved Hide resolved

robinostlund added 2 commits November 28, 2023 16:33

fix black

79e3f49

fix black

fbd8b5b

many properties fixed, some new TODOs :)

3b21d0b

github-advanced-security bot found potential problems Nov 28, 2023

View reviewed changes

volkswagencarnet/vw_vehicle.py Fixed Show fixed Hide fixed

fixed unused variables

9699332

Merge branch 'master' into rework-api-after-carnet

d148f4d

some cleanup of comments

d94b545

den4uk reviewed Nov 28, 2023

View reviewed changes

volkswagencarnet/vw_connection.py Show resolved Hide resolved

den4uk reviewed Nov 28, 2023

View reviewed changes

volkswagencarnet/vw_vehicle.py Show resolved Hide resolved

den4uk reviewed Nov 28, 2023

View reviewed changes

oliverrahner added 4 commits December 5, 2023 22:58

fix parking time bug when moving

e77839d

remove vehicle_moving entity

d3f69de

begin force wakeup

97fa4e0

only update parking position after last trip changed

8fedaa8

oliverrahner force-pushed the rework-api-after-carnet branch from 6d12a41 to 8fedaa8 Compare December 5, 2023 21:59

oliverrahner and others added 2 commits December 5, 2023 23:12

added some response examples

bba0129

Fix Re-Login race condition (#223)

dfa2f25

oliverrahner closed this Dec 18, 2023

rework authentication and API structure after VW has switched off CarNet #216

rework authentication and API structure after VW has switched off CarNet #216

Conversation

oliverrahner commented Nov 28, 2023

robinostlund commented Nov 28, 2023

den4uk commented Nov 28, 2023

oliverrahner commented Nov 28, 2023

oliverrahner commented Nov 28, 2023

den4uk commented Nov 28, 2023

TheBlackBGD commented Nov 28, 2023

LucTs commented Nov 28, 2023

robinostlund commented Nov 28, 2023 • edited Loading

TheBlackBGD commented Nov 28, 2023

virtualdj commented Nov 28, 2023

oliverrahner commented Nov 28, 2023

TheBlackBGD commented Nov 28, 2023

oliverrahner commented Nov 28, 2023

TheBlackBGD commented Nov 28, 2023

den4uk Nov 28, 2023

Choose a reason for hiding this comment

oliverrahner Nov 29, 2023

Choose a reason for hiding this comment

oliverrahner Nov 29, 2023

Choose a reason for hiding this comment

stickpin commented Dec 5, 2023

oliverrahner commented Dec 5, 2023

oliverrahner commented Dec 5, 2023

stickpin commented Dec 5, 2023

oliverrahner commented Dec 5, 2023

oliverrahner commented Dec 5, 2023

stickpin commented Dec 5, 2023

virtualdj commented Dec 6, 2023

stickpin commented Dec 8, 2023

oliverrahner commented Dec 8, 2023

oliverrahner commented Dec 8, 2023

stickpin commented Dec 8, 2023

stickpin commented Dec 8, 2023 • edited Loading

stickpin commented Dec 8, 2023

stickpin commented Dec 8, 2023

oliverrahner commented Dec 8, 2023

stickpin commented Dec 8, 2023

stickpin commented Dec 8, 2023 • edited Loading

stickpin commented Dec 8, 2023

stickpin commented Dec 8, 2023

robinostlund commented Dec 18, 2023

oliverrahner commented Dec 18, 2023

robinostlund commented Nov 28, 2023 •

edited

Loading

stickpin commented Dec 8, 2023 •

edited

Loading

stickpin commented Dec 8, 2023 •

edited

Loading