Delta 2 (MQTT) #6
Comments
|
I think I've come up with a sort of "work around" for this type of messaging that is similar to what @lwsrbrts came up with for DP and I adapted for use with SHP... Like SHP (and unlike DP) the Delta 2 messages have an identifier... For SHP this is an "id" in the payload that is unique to the module in question... for D2 this is a "moduleType" in the payload... Created the following automations to re-publish payloads with certain module types to new topics: BTW: I assume moduleType '4' is for the extra battery but I don't have one to prove that... It's also possible that could appear under moduelType '2' as, unlike DP, both 'bms' and 'ems' keys appear under this same moduleType... Then, created MQTT sensors to pull the above into sensor attributes in HA: Then, not knowing which keys will appear in a given message payload for a given moduleType, I came up with this workaround to build out custom sensors for specific data points: ... If the given attribute is not available in a given message the sensor stays set to it's last known state but if it is available (and not 'none') the state gets set to the value of that attribute (with whatever modifications I want like taking the voltage in mv and converting it to volts). Still a bit tedious to identify which data points and build out sensors for each but at least it looks like this can now be done... It would certainly be nice if they would just give us a proper, locally accessible API or a secure way to enable a local MQTT broker on these devices... This is way better than nothing but I'm no fan of having all this tied to the cloud and it's starting to feel like a lot of layers and latency... |
|
I'm exploring a different approach which may work as alternative for D2 (as well as DP and SHP - though the payload for 'lastestQuotas' on SHP is HUGE)... In the case of all 3 devices (SHP, DP, D2) when the app opens it publishes to ../get requesting 'status of everything' like this: The device will respond by posting a large payload to ../get_reply with JSON containing the status of 'everything'... In this alternative approach, I created two automations. One that triggers every 10 seconds to ask for status and another to grab the response from ../get_reply and post it to ../status I then created an MQTT sensor that sets state based on the value of the 'online' key and pulls the rest of the payload from 'quataMap' into attributes of the sensor... At that point all the data is in HA and is updated every 10 seconds. You can display select attributes in the Templates section of Developer Tools and use them as the basis for custom sensors, triggers, etc. The following: Should produce output like this: |
|
I feel like the above approach is a bit cleaner than constantly triggering multiple automations on constant messages flowing into .../data and having to re-publish those to various other topics and then have an MQTT sensor for each topic... However, it does mean updates are limited to how often the automation in HA triggers to ask for 'status' (every 10 seconds in my example) and does turn this monitoring into more of a 'polling' operation... With this approach you don't need ../data in mosquito.conf but instead need: I'd like to get some feedback on this approach (at least conceptually if not specific to the Delta 2) vs parsing each inbound message to ../data and re-publishing to various other topics before pulling, separately, into HA and also any opinions on if 10 seconds is too often or not often enough, etc. |
|
While testing this alternative, I also played with having the automation that publishes ../get 'status of everything' trigger on any message published to ../data In this case updates are constant but so are requests asking for 'everything' - the rationale is that if something got published to ../data then 'something' changed so let's update 'everything' (to our single sensor in HA) and go from there... the potential drawback I see if that every time the device publishes an update to SOC, for example, HA will publish a request to mosquitto, which forward to public MQTT, which is picked up by the device, which then publishes a full payload with 'all status' that is then forwarded to Mosquitto read by HA, republished to Mosquitto under ../status and then read, again, by HA into the MQTT sensor entity... In the case of DP, all the keys for status of a given module (bmsMaster for example) are sent together for each update so while you could use this alternative for DP is somewhat less compelling as you end up with 'all status' for each in one of 5-7 MQTT sensor entities in HA (though it still could be nice to have them all in one). With D2 it's a bit more convoluted since any given message contains only changes of status for 1 or a few attributes (simpler to handle in a purpose built script, more convoluted to get into usable form for HA)... I have not decided which approach I will ultimately take in building out custom sensors to monitor D2 and I have yet to work on using HA to make changes to settings by publishing payloads to ../set Ultimately all of this would be more elegant built as a formal HA integration but I'm not sure I'm ready to think about tackling that at this point... |
|
Just a quick update to say that I've refined my D2 integration further... I ended up going with 2 automations for MQTT such that one publishes a requests for 'state of everything' to ..\get whenever 'any' state change is published to ..\data (by the device) and another that triggers on receiving the list of all states on ..\get_reply and re-publishes the payload containing 'latestQuotas' to ..\status I've also got a number of MQTT sensors setup (working from data in ..\status) as well as automations that update input_numbers and input_lists based on various payloads received on ..\set or publish appropriate payloads to ..\set to change settings (on the device) when inputs are changed on the HA dashboard... I do have a bit of an issue to figure out though where changing a setting in the app results in the automation to change input on the dashboard executing which then triggers the automation to publish the same change to ../set (because the input on the dashboard changed)... At this point it just causes the device to beep twice because it was told to change the setting twice (by app and HA) to the 'same value' (if beep is turned on)... but it "works"... I just find it annoying because it is ugly/kludge and I want to figure out how to employ some kind of logic to prevent redundant commands to change the same setting to the same value... Here is what I have working, thus far in HA, for Delta 2
|
|
Hey, I would like to combine these information with an external solar generator and a zigbee socket to charge Delta 2 efficiently. |
Here is what I did to trigger a publish to ../get when a change is published to ../data and also to re-publish the response on ../get-reply to ../status -- at that point you can build out MQTT sensors using ecoflow/D2/status as the topic... I'm still working on building out sensors and refining automations... but here is an example MQTT sensor that uses the data published to the ../status topic to pull SOC into HA... |
|
Thank you! the automations and mqtt senosr to get the status are working like a charme! |
I am working on refining the process to control and change settings... it's a matter of capturing appropriate payloads from ../set as you change settings on the app and then publishing them (potentially with some alteration) back to ../set from HA using automations and input_numbers/lists, etc. Right now I have an issue where if I change something in the app and my automation then changes the slider in HA, then another automation that works in reverse kicks off to change it (to the same thing) by publishing the same command again.. I'm, not sure, yet, how to prevent the double trigger without shutting down the ability to change settings from HA... |
|
just played around to change some values (e.g. input power) and i´m facing a problem with publishing in /set. { so i have to switch to another value and back again in hope the change will be applied. |
I encountered some oddities in the beginning and have adjusted a few things and, at this point, don't seem to have an issue...
I don't know for certain if all of the above are "required" (though you will have issues for sure if your remote_clientid is not unique)... But it does make it much easier to troubleshoot if you're watching the broker and can see if a command came from "Android", "IOS", or "HA" (what I use to designate 'home assistant') and, at one point, I suspected sending every payload with the same "ID" (like '999999999') was causing some issues (at least for Delta 2) which may or may not be an issue for every device and/or setting (depends on how the device itself actually interprets these payloads)... You'll notice in one of the examples above I use the following for the payload: This means each time the payload is posted by HA to ../set it will be "FROM: HA" and use a unique ID between 999910000-999999999 Hope that helps... |
|
point 2 and 3 was already implemented in my config. in MQTT explorer I can see the changes sent, but it will not be applied..idk whats wrong now EDIT: found it! should be: |
Yup... that will do it... set to "in" the payloads published to mosquito from HA won't be sent to mqtt.ecoflow.com... |
|
NOTE: I made a final attempt to move away from constantly publishing a request for 'latestQuotas' to ../get and then re-publishing the response (with 'state of everything') to ../status for use by my MQTT sensors... While this does "work" the problem is that D2 only publishes the state of most things when/if they change so you end up with a lot of sensors in "limbo" until everything on D2 has been exercised... This makes for an ugly dashboard full of gauges in "unknown" state... I'm not sure how "inefficient" it might be or how much (if any) extra power I'm using by querying for 'state of everything' every time any change is published to ../data (/app/device/property/SN) but the result is having all sensors constantly up to date and not having any in "unknown" state when sensors are reloaded and the dashboard appears quite responsive... ... still feels a touch kludgy or somehow "inelegant" so if anyone else comes up with a better way to handle this for Delta 2, please let me know! |
|
Are you able to control the fan of DP2 ? |
I found that as well... With the most recent firmware I see those two keys in both 'latestQuotas' and sometimes in the payloads published to ../data but, thus far, I've only ever seen them be zero no matter what the fans are doing on the Delta 2... As far as manipulating the fan speed, I doubt that is possible... There are no settings in the app for any of the devices I'm aware of to do that... I do have a sensor to display fan speed in HA for Delta Pro (which works) but can't "control" the speed... For D2, thus far I have not found any place that the unit actually updates to say what speed fan are running at or even if they are running at all... If you or someone else discover something, please let me know. I would at least like a sensor to monitor fan speed over time and be able to compare that with temperatures, loads, etc. |
|
yes... |
|
Which firmware version do you have on delta2 ? |
|
I'm running the most current 1.0.0.64 firmware on Delta 2 with WiFi firmware of 1.0.3.18 and that is what I used to develop my MQTT package and dashboards for HA... |
|
Thank you ! I have just updated it to the newest version, and it works very well EDIT1: after the firmware upgrade is : working very well! I used before: because the values were not updated so quickly on version 1.0.2.0, but it was not satisfying EDIT2: this change causes a higher number of write attempts --> wearout of SSD/HDD an a higher CPU usage |
This is good to know... I'm running HA in a VM on a large Hyper-V server so I'm not too concerned at this point... however, I still feel my approach of constantly asking for "state of everything" is a bit of a kludge... Perhaps someone can suggest a better approach... The main issue with just pulling from ../data is that the state of many sensors are not updated frequently (if at all) via that topic so anytime sensors are reloaded a bunch of them can remain in 'unknown' state for a protracted period of time... |
|
maybe this issue can help to improve your version ;) |
Interesting.... I'm not sure I totally understand how to implement that but, assuming the data/topics are laid out the same and messages are reported the same as using the public MQTT, I'm not sure it resolves the challenge with regard to pulling this into HA... Perhaps there is a way, in Home Assistant, to call a script or trigger an automation that pulls the comprehensive quota data "once" and populates the "current state" when/if sensors get reset (or HA is restarted)... I'd have to dig back into it but there may also be some states that never get updated (though I think most, maybe all, do update when/if changed)... |
|
Just install the android app and put in your wifi-password and the ip of your mqtt server. its working on port 8883, so there is some config needed, if not already done. Wifi-Reconnects seem also gone away. for now I tested some commands like switch USB on/off etc and it responds so much better and faster, then with official mqtt server. The "old" commands are working fine here.
States of some switches are missing here. (e.g. USB switch//dc24vState) |
|
here are some entries from my configuration.yaml : Problems:
|
Are you talking about the Ecoflow App itself or one of the other apps mentioned in the thread? And are you saying mosquito needs to be reconfigured to operate on port 8883, over TLS, instead of 1883 (no TLS) as I currently have it setup (by default)? And by WiFi-password are you referring to the password for the ESSID (wLAN) the D2 is connected to, the credentials for the EcoFlow login (from the native app) or the password for MQTT credentials? Might you be willing to share a redacted version of your mosquitto.conf? Sorry if theses questions seem dense. I'm not that familiar with Python or Android development and haven't really had time to dig into this approach in depth. I read through the linked thread one time and didn't entirely follow the process to get D2 connected to mosquito. Once I've got that accomplished I think I should be okay as far as adapting my HA packages to work locally... so if you can assist me with the shortest, clearest path to that end it would be greatly appreciated. I'll be happy to share the adapted YAML, etc. on my repo once I've got a handle on this approach... Hoping the same will work for SHP and DP as well... Also: I'm assuming once D2 is talking to local MQTT broker (mosquito) it could then be bridged to the cloud MQTT if one wanted to allow for remote control or sending data to EcoFlow (basically the reverse of what we did in the first place to bridge public MQTT to private broker) ?? If I can gain 100% control locally via HA of all my stuff I'm not sure I'd ever choose to connect with the cloud server any longer. However, you never know if support, updates, etc. may be needed and there might be a need to either reset to "normal" or pass data through... In reading through the thread it sounds like certificate checks may have to be bypassed and probing for firmware updates "faked" or "ignored" ?? |
|
here is a little manual:
Details
here are all of my sensors from configuration.yaml that i´m using: Details
if you open the official app, Delta2 will connect to the official server again and the procedure from point 3 needs to be repeated.
you could try it, if it also works with SHP. I have still no SHP at home, why I could´t test it :)
ipalchuk mentioned it in his thread, that it could be better to bridge the connection to official mqtt server
that the same thing I think. For my usecase I don´t need the official server and the official app at all.
thats also a point ipalchuk mentioned. i´m running this configuration since yesterday and yes the device is sending periodicaly his module versions etc., but I didn´t see any problems so far. if problems will appear due to this, we need to write a script for HA to answer some device requests. |
|
@jegres1709 thanks for the detailed summary and example YAML... I'll start playing with this approach as soon as I can find some time... One challenge I have is that my Android "devices" are virtual (emulation on PC) so I don't have a ready way to attempt this over BT... I'll have to see if I can side load that apk on an old fire tablet or something... If just opening the native app resets the device to point back to the EF cloud server for WiFi that means that once this is setup you need to transition to using HA exclusively. That could be a pain for updates and native testing of new features... I'm wondering if there is some way to have HA "reset" this pointer under given conditions or even periodically... I can just see opening the native app to check something and then losing all subsequent data into HA until this is detected and reset... I suspect such a solution would involve using BT capabilities in HA (which would involve additional hardware for me since my HA instance is in a VM). One interesting aspect of this is that EF could (if they choose) supply a local option for the official API that uses the same/similar set of MQTT/JSON topics/payloads as the cloud approach. An API that exposes the data we all care about the most that is accessible locally without internet or cloud server (and fully sanctioned by Ecoflow) is, I think, what everyone wants (at least those of us integrating with HA if not many others that would prefer a non-cloud-centric approach). |
|
Hello, I am absolute newbie on github and no native speaker/writer .... sorry for my english :) ... |
|
Can I please get access too @Ne0-Hack3r? |
|
Hello @Ne0-Hack3r?, These ar the loggs in mosquitto-broker: UPDATE: Thanks to @jegres1709 and his description I finally got it! |
|
@Ne0-Hack3r, can I please get access too to your repo/scripts... will start investigation for Delta 2 (Max and maybe Powerstream later). |
@jegres1709 Könntest du mir bitte deinen Code zusenden? Hast du auch eine Möglichkeit gefunden, das Display zu dimmen? |
@jegres1709, I am interested in the code, too ;-) |
|
sure. dashboard looks little different now. You will also need some HACS lovelace extension like "vertical-stack-in-card", mushroom-chips-card.
Details
|
Thanks, I will give it a try when my Delta 2 Max arrives and start looking in next step for Powerstream ;-) @jegres1709, do I still need the access to the repo from @Ne0-Hack3r ? Unfortunatelly he does not respond, do you have the needed files/scripts/code elsewhere? |
|
little offtopic:
Details
|
code for what? the code for the dashboard is in the "Details" |
Thanks, yes I think that is the part where I need to dig deeper after I have my Delta available. |
Done |
|
Done |
|
Hey @Ne0-Hack3r , could you send me an invite. I'd really like to have my delta connecting to a local mqtt. I have major connectivity issues where I'm located typically off grid so has integration becomes a pain. Thanks! |
|
@radeonorama - invite sent... bear in mind that Delta 2 and Delta 2 Max are the only Delta models that can be hacked, over BLE, to do local MQTT and that process now requires a BLE auth packet to be formed and sent upon connection. The Delta Pro, Smart Home panel (and I presume the Delta Mini/Max) communicate with the back end over TCP via byte stream which, without an open port, is a problem. I've still found no way to gain 'local' control of my SHP (only MQTT via the cloud server). |
|
Thanks v much for this! The dashboards you have made available are amazingly detailed and kick the sh*t out of the EF official app data! I just need to find the time to do the same with the wave 2! |
|
Hey @Ne0-Hack3r , could you send me an invite. |
|
@Yrkis - done. |
[...]
I'm still VERY early in the process, but this is so far matching up very well with my newly acquired River 2 (base model) unit. And yeah...it seems VERY different from my Delta Max 2000 (original, not "2") which closely followed the Delta Pro implementation |
|
Does the Delta2 take a MQTT message for setting the RTC from the app? I'd say 90+% of this Delta 2 thread has applied to my River 2, but I'm stuck with the message where the app sets the clock which seems to be potentially tied to getting correct responses back from the River 2...maybe related to its schedule/automation portion of its responses?
Specifically there is a "week" field that was "1" on December 2nd and is now "7" on December 3rd. Its clearly not "week of year" nor "week of month", and I am unsure what it might be. |
@mmiller7 the "week" is "day of week" but is a bit unintuitive... I have not found it to be necessary, thus far, with D2, D2M, or DP to post this RTC payload... For D2, data is continually posted to the /app/device/property/{sn} topic with status of sensors (as they "change")... You can still post to ../get for 'latestQuotas' and a paylod with status of all sensors will be posted, by the device, to ../get_reply I am not on the latest firmware for D2/D2M as I hacked mine over BLE to report directly to my local MQTT. The prior update added a BLE auth packet that had to be reverse engineered and this next update may also add some form of encryption (at least for D2M)... seems like it is getting harder over time to keep up this reverse engineering game... |
Odd, mine doesn't seem to do this unless the app is open. I'm wondering if this is a firmware change, since my Delta Max 2000 used to publish on-change to the data/property one but after an update last night no longer publishes unless the app is open, which is the same behavior I see with the D2. I'm not using Bluetooth, just WiFi thru their servers...it does seem like things are changing a fair bit over time. Might have to modify mine so that it attempts to query using the latestquotas...I was hoping to minimize spamming their servers by letting the data flow in on its own. |
@mmiller7 - If you are using the native app and then closing it that will keep future updates from posting to the broker unless you restart the broker after the app is closed. |
|
@Ne0-Hack3r, would you please give me access to your repo? I have been looking for accessing Ecoflow Delta 2 for a long time. You've done an amazing work!! Thank you! |
|
Done... Note that https://github.com/tolwi/hassio-ecoflow-cloud is full HACS integration that has support for Delta 2 as well. Also, be aware that some of the hacks to obtain local control of D2 and D2M may no longer work with later firmware so cloud based MQTT may be the only solution moving forward. I have not had time to look into what roadblocks new firmware introduces on the BLE side or what it might take to overcome those (assuming it is even possible). And, finally, you may also want to know that the official EcoFlow supported API is now public and includes Delta 2 as well. More info here Facebook group here |
|
@Ne0-Hack3r Thank you very much! |
|
@Ne0-Hack3r - can you please provide access to your repo? I'm working on setting up my SHP + dual DPs + HA. Thanks. |
|
@Ne0-Hack3r Could you please invite me to your repo? Much appreciated, thanks! |
I've started down the road of trying to integrate Delta 2 data with HA via MQTT...
I already have mosquito broker setup as per the instructions from @lwsrbrts in ISSUE #1 and my tweaks for SHP per ISSUE #5 so I simply added the following to mosquito.conf:
After seeing the ../D2/data topic on the local broker via MQTT Explorer I used a power shell script to gather messages and log them over the course of several minutes and captured a number of short messages with 1-5 key/value pairs each similar to this:
After parsing and sorting a bit I came up with the following list of "unique" params:
I also grabbed the payload from ..\get_reply in response to "lastestquotas" from the EF MQTT server (sent to ..\get by the D2 app when it opens) and discovered these unique keys:
It appears the D2 works differently than the DP. The DP will send a large message with the status of 'everything' for a given module (bmsMaster, ems, inv, ...) on a fairly frequent basis. SHP appears to operate in a similar way. The D2 appears to send only small messages with what seem to be status "updates" for specific data points which would seem to imply the app gets the status of 'everything' when it opens and then only receives update messages past that point... I'm still new to HA and MQTT and I'm not an "app developer" so some of my speculations here (based on initial observation) may well be incorrect...
For DP and SHP we can create automations in HA to trigger on specific payloads for specific params in messages such as "bmsMaster.errCode = 0" (for DP) or "id = 2" (for SHP) and, based on those triggers, re-publish the entire message to a unique sub-topic (such as ../SHP/circuits) then use an mqtt sensor to pull all the key/value pairs into attributes of an entity in HA. Once those attributes exist in HA custom sensors can be made to track specific attributes (such as sensor.shp_circuit_1_power) or the attribute values can be used directly in various places like {{ state_attr('sensor.shp_circuits','infoList')[10].chWatt | round(0) }}
For D2, the messages arrive with a unique "moduleType" but may contain one or several different parameters... The moduleType could be used by an automation to re-publish to a unique topic per module... but at that point I'm a bit lost in terms of how to structure mqtt sensors to pull the specific data/attributes into HA...
Perhaps someone here with more HA/MQTT background can suggest the best approach?
The text was updated successfully, but these errors were encountered: