-
-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add an API endpoint for upstream update URL retrieval #4470
Comments
@fonix232 thanks for your proposal! I agree that - especially with HA users being quite relaxed about "letting HA pick and install updates for me" - the current approach of HA will end in a usability disaster, and lots of devices going offline (running bootloops because they were flashed with the wrong firmware update). It might be possible to add new json endpoint like Not sure if we would ever manage to get all usermod authors into the same boat, and extend your idea to usermods. In fact there are >30 usermods right now, that can be combined arbitrarily - each combination is a different firmware.bin file. afaik @Moustachauve has asked for something similar a time ago, and we agreed that WLED_BRAND and WLED_PRODUCT_NAME could be used to help 3rd party tools understand if its "vanilla" WLED or a fork like MoonModules WLED. Lines 805 to 806 in 709aeff
This is also implemented in MoonModules WLED - I can't speak for other forks, like WLEDtubes by @SteveEisner and @craiglink. WLED-MM has The current solution only allows to identify the fork, however its true that each fork has own naming conventions for firmware files in the "release" section of the repo. We always wanted to find a way to atomaticially mark custom builds as "custom", to prevent friendly HA from overwriting it with a standard build from another repo. But this has not been implemented yet - we are currently struggling with more basic things like nightly builds using github CI actions (github API's were changed several times during the last year).
This sounds like a bug in the HA update procedure - We (MM team) actually changed our versioning scheme with the last release, in particular to stop such crazy update notifications. WLED-MM latest version is "14.5.0-beta" without the leading "0.", so HA should not try to overwrite a "14.x.y" with a "0.15.z" from another repo. |
PS: i have some boards at home running custom builds of ESPHome, and for some reason HA is able to distinguish custom builds (don't update) from vanilla standard builds (offer updates). Maybe we could learn from ESPHome? |
FYI WLED's {
"info": {
"ver": "0.15.1-b1",
"vid": 2501090,
"cn": "Kousen",
"release": "dev_release",
...
"fs": {
"u": 102,
"t": 983,
"pmt": 0
}
...
"arch": "esp32",
...
"u": {
"Temperature": [...],
"PIRsensorSwitch": [...],
"MultiRelay": [...],
"AudioReactive": [...],
...
}
...
"brand": "WLED-BB",
"product": "Srg74-Wemos",
"mac": "3c6105XXXXXX",
"ip": "192.168.XXX.XXX"
}
} IMO plenty of information to decide if correct firmware (from official repo) is available. Adding custom repositories (as would be needed for above example) is beyond WLED unless auto-update is implemented in WLED itself. |
.... and here comes the same info from MoonModules WLED: {
"ver": "14.5.0-beta",
"vid": 2501070,
"release": "esp32S3_16MB_PSRAM_M_HUB75",
"rel": "esp32S3_16MB_PSRAM_M_HUB75",
...
"fs": {
"u": 8,
"t": 10354,
"pmt": 0
},
...
"arch": "ESP32-S3",
...
"clock": 240,
"flash": 16,
...
"psram": 8245467,
"psrmode": "🚀 OPI",
...
"u": {
"Autosave": [...],
"AudioReactive": [...]
},
...
"brand": "WLED",
"product": "MoonModules",
"mac": "dcdaXXXXXXX",
"ip": "192.168.X.X"
} So yes, lots of info to decide where to locate updates; the most important fields are aligned across our forks already:
So putting it all together --> I don't fully agree with @blazoncek, because it would be simple to add one extra attribute to info that identifies the URL, like "repo": "https://github.com/MoonModules/WLED/releases",
"file_prefix": "WLEDMM", we could automatically populate this field from the downside is the string occupies ~60 extra bytes on flash, which is a very tight resource already. |
@softhack007 I said "unless auto-update is implemented in WLED itself". This does not exclude implementation. And BTW there is intent on doing that. |
I tried to start similar fix for HA to be repo aware but blocked at the first hurdle by the author of python-wled If we try your model of per-device repo URL for each WLED device rather than selecting your repo on HA preferences, then we would still need to either create and maintain a fork of python-WLED and puruade HA to use that or trying and puruade the author to reconsider |
... as @frenck is also the maintainer of the HA integration for wled, I think the outlook for making this happen is not good 🤷♂️ at all. Edit: @netmindz your short discussion with him sounds like HA has no interest to even think about forks that are not OTA compatible with AC WLED |
It would be better to read this from git remote info rather than a value that needs editing when people create their own fork
You only need to know the namespace and project, e.g "AirCoookie/WLED" as the rest can be hard coded prefix and suffix in the python script used in HA or on in the mobile app |
From his limited feedback it felt more like he just didn't want to open the can of worms of doing anything at all with anything that isn't AC - just in case there are differences in the actual API in those forks. What I'm asking for and what my PR does is nothing to do with the API, just allowing a hard coded value in the library to be one that can be overriden, so doesn't really change the likelihood of people reporting issues with the library being used against anything other than AC |
If you will be adding this field to JSON API be sure to measure generated JSON size on ESP8266 and most used usermods included.
I totally understand that. It would be a nightmare from the support perspective otherwise. That is also why I was pushing so hard to get many usermods included by default.
This is not entirely true. If you try to OTA S2 binary to S3 it will be unsuccessful and will not render the device inoperable. However OTA S3 OPI to S3 QSPI device will brick it. |
I strongly disagree. This isn't a value used by the firmware itself, but rather - just like the other identifiers used by usermods/forks - a field that says "hey, I came from here, check here for updates". WLED itself doesn't need to implement auto updates, that's down to the client, but for the client to provide the right update, it needs to look in the right places for the update in the first place. Since neither the brand nor the product nor the release field tell the client where the updates reside. Adding the extra field has absolutely no negative effects, and usermod developers can even ignore them (in which case the clients can easily fall back to the current approach of checking for updates through the hardcoded URL), and adding them is quite literally just slapping a compiler parameter in the build configs (let it be the platformio INI, or a GHA workflow call to
Bit off topic, but ESPHome actually does offer updates for custom builds. In fact, aside from the basic Tasmota-style firmware ESPHome offers (the "initialisation build"), every single other configuration is custom and requires a custom firmware compilation (which the ESPHome addon takes care of). ESPHome has its own issues with this too (e.g. there's no way to create a "generic" firmware you'd flash on multiple devices of the same configuration, where the only values changing would be the device name/ID, and network stack, so if you have 12 of the same device with the same ESPHome config, you need to compile that 12 times), but this is actually something they do right.
There's actually another bug with this approach that I felt wasn't serious enough to warrant a bug report/feature request, as it only affects a small subset of devices: many older (ESP12/8266 based units mostly) can't handle the larger "full size" update BINs and need to rely on GZIP'd files. I have three such controllers in my home right now (Mostly Athom and Gledopto), and it's always a fight for them to get the right file. Again strafing off topic, but maybe it would be worth adding a field to the
I think we need to be cautious with adding these extra features on firmware side. As pointed out above, WLED is already growing beyond what some older models are capable of (storage, computing performance and RAM bound), and given there's plenty of off-the-shelf models being sold with those older chipsets, dropping support for those in favour of built-in auto OTA is something many people would not be happy about. Of course I have much less visibility of this given my limited number of devices and models, so I refer to the team's (and usermod devs') expertise and experience here, but I felt it needs pointing out this possibility.
Given that the update checking logic would be GitHub bound anyway, these 60 bytes could be reduced to Another potential approach would be enforcing that usermods follow a GitHub naming convention of
@netmindz the issue with this, as I pointed it out in my original post, is that it puts ALL the responsibility on the client developers. Right now they already have to maintain the auto OTA feature, which is hard enough, but having them monitor forks/usermods and keeping an up to date list of product name to repo mapping, is simply not tenable. By pushing the update repo location into the firmware, this check becomes as simple as:
The logic change here is as simple the first if/else block instead of the fixed repo. Minimal change, yet opens up OTA functionality to most forks/usermods. @frenck was right to refuse your original idea as that gives him an immense amount of extra work not just once, but also through continuous maintenance. By lifting that out and putting the responsibility on the fork/usermod maintainer, I'm fairly certain he'd be agreeable to the above outlined pseudocode'd change.
I disagree, Frenck in the past has been quite open to sensible changes. But having the client developers deal with all the possible forks, with absolutely no help coming from the device itself, that simply doesn't make sense. Why should the CLIENT be aware of all the forks, when it's absolutely not necessary, and the firmware itself could report where it came from.
To be fair, it would introduce a fair bit of complexity to be considerate of forks on their end. At the end of the day, anyone's free to fork both python-wled and the WLED integration, and add their own code. If I wanted to, I could have added the MoonModules support to python-wled and gotten my needed path of auto upgrades with much less trouble.
At the end of the day, it's up to the user to ensure updates are correct. HA only offers an update, and if it's incompatible, you can always disable it. But yet again I have to repeat myself, WLED at the moment offers no way to determine where a non-AC variant came from, and pushing the responsibility on the client devs to maintain a list of all active usermods, repositories, update approaches, is simply not viable. With WLED providing that info and usermod makers sticking to a good pattern in naming files, etc., support can be added without the need for continuous maintenance. |
I also disagree with that statement and for the same reasons you list
The gz version is it's own challenge.
Aside from the gz issue there can also be a number of reasons why the partitions are not the expected size, so telling the user why OTA is not possible is an increasingly big issue. That should be tracked it's own issue. Please create a new one to cover both
Due to resource constraints it's never going to happen within WLED itself, however as part of looking at the fact we are out of space, we are looking at possible alternatives to the current OTA process.
Hence my comment saying why we have no reason for the full release URL ;)
I don't think usermods is really the right term. What you appear to be talking about is forks. That's why I said it's better to be automated based on the git repo, not relying on Devs to manually change any of the source
I think you misunderstand me. In the first instance, allowing the library to query GitHub for releases from a repo of your choice, not hard coded to AirCoookie/WLED does not put any responsibility onto the author/maintainer of that library. In that instance if I know say all my devices run my own build. I just set that repo in my app preferences. That will help in many cases but not all. The more complete solution, which I illuded to in the python-wled PR, is that we flip it round as you are also suggesting whereby the client asks wled for the repo name. This then removes the need for the user to set the update repo in their preferences and also allows for users that have devices running firmware from different repos. This puts more work on the client developers as they now need to poll every device and then fetch the release info for that repo and cache the value so you aren't searching release info multiple times for the same repo. Not a massive change, but more work than I was proposing as the initial phase of work.
Correct
That isn't taking account of pre releases, but I get the general point you are making
See earlier comments regarding the swap to per-device lookup and caching
As mentioned, there is zero maintenance required if you simply give the user an option text box they can populate themselves
If you use my PR you are half way there. I just don't have a PR yet to allow you to define your custom repo name in HA
My proposed first step put the responsibility onto the user to set their preferred repo. I do agree that pushing the responsibility to the author of the firmware is preferable, it's just how to make that something easy for them to do and also I the extra complexity on the client side to handle per device release info. How good is your python? Can we perhaps resolve this by you talking on the task to create PRs that handle the changes to HA to support per-device repos? |
wow, long discussion :-) Just one point for clarification:
This is not entirely true - yes positively identifying the "correct fork" and repo is not feasible atm without changes in WLED. This "custom fork check" could be implemented easily in HA today. And it could be relaxed later, once we have found a way to tell the integration where to look for updates. |
The esp8266 chip with =< 2MB flash (including boards like esp12F, esp01, esp01-S, nodemcu), is the only device that requires a compressed binary for updating. esp32 boards do not even support compressed firmware uploads. I see two easy options to solve this on the integration side:
|
There appears to be something in the code already to handle the 2M devices |
We'll incorporate whatever approach y'all decide. It's our goal to minimize differences in our branch, so we just sync to main (going to do that soon for 15.0). BTW I tried writing some very simple code for updating units -from each other- but found that for some reason the web stack didn't stay alive long enough to allow the necessary multi-MB firmware transfer. I haven't spent much time on it or gone back to look at it for a while :) https://github.com/SteveEisner/WLEDtubes/blob/main/usermods/Tubes/updater.h#L305 |
Did you see this comment @fonix232 ? |
I didn't read 100% of what was listed, but I glanced over and wanted to say what my plan was for WLED Native auto update feature. I wanted to have a manual map of I then use the approach listed above by @softhack007 :
Indeed, that's what I do with WLED Native (Android). I think HA should definitely do at least that. |
Is your feature request related to a problem? Please describe.
With all the usermods floating around, it's getting harder and harder to track which specific build one needs and how one can check if there are updates. Many clients (incl. wled-native, the Home Assistant integration, or anyone utilising libraries like Frenck's python-wled) have now built-in update pathways, however these are currently hard-coded to using the main repo - not exactly optimal to have HA or wled-native prompt you to update your device to a potentially incompatible build. An example would be my neat little 64^2 HUB75 matrix for which I'm using an Adafruit MatrixPortal S3, and the MoonModules fork that supports the protocol - which is still on 0.14.5, yet I'm getting constantly notified that there's an update.
Describe the solution you'd like
Add a simple API endpoint that returns a hardcoded value set during compilation, so that forks can provide their own GitHub address, which in turn can be used to check for new releases and updates.
Describe alternatives you've considered
I initially thought of adding fork detection + hardcoded update paths to the clients themselves, however upon seeing frenck/python-wled#1597 (which, based on Frenck's comments, I believe was the same core idea as mine), I realised that that idea pushes the responsibility of maintenance of the client developers. By having the firmware+device combo be able to tell "hey I'm this fork, and you can find updates for me [here]", this issue is alleviated, and is the superior solution.
Additional context
Obviously this would be a change that needs to propagate to the forks and would only become useful once widely adopted - therefore I think the sooner this gets pushed to
main
, the better.On the technical side, this is a relatively straightforward thing to add, with the following steps:
WLED_OTA_REPO
)wled00/json.cpp
'sserializeInfo
method, add a new entry to theroot
JSON object:And from this point on, any client library can utilise this field's value to check for OTA updates instead of relying on hardcoded URLs.
Caveats:
The text was updated successfully, but these errors were encountered: