Add MQTT dispatcher #220
Add MQTT dispatcher #220
Conversation
|
OMG, watching intently :) |
datastore/mqtt/mqtt.go
Outdated
| var tokens []pahomqtt.Token | ||
|
|
||
| // Publish each field to the relevant topic on the MQTT broker | ||
| for key, value := range convertedPayload { |
There was a problem hiding this comment.
instead of sending each payload attribute individually, why not send it as a single message and keep the implmentation as other datastores?
There was a problem hiding this comment.
For MQTT topic structuring, it is important to maintain clarity and scalability, especially when dealing with dynamic datasets such as fleet telemetry and event logs.
Sending individual metrics over MQTT rather than combining them in a single topic has several advantages:
- Granular control: Individual topics allow for more precise subscription and filtering. Clients can subscribe only to the specific metrics they need, reducing unnecessary network traffic and processing. A Power Exchange (https://pex.energy) will likely subscript to SoC fields, and a Home Assistant (https://www.home-assistant.io/) might subscribe to location data.
- Easier parsing and debugging: With separate topics, the payload structure for each metric can be simpler and more consistent. This makes parsing and processing easier on the receiving end.
- Scalability: As the system grows, it's easier to add new metrics without modifying existing payload structures or breaking existing subscribers.
- Retention policies: MQTT brokers often allow setting different retention policies per topic, giving you more control over how long each metric's data is stored.
Here is a screenshot from MQTT Explorer demonstrating a clear and scalable topic structure:
| } | ||
|
|
||
| // Only process reliable ACK if no errors were reported | ||
| if !publishError { |
There was a problem hiding this comment.
yeah this is tricky with the things you have correctly set up. you are asking the vehicle to resend the whole data if partial uploads to mqtt failed. I think we should just attempt to send the whole payload
There was a problem hiding this comment.
See my comment above which explains why the MQTT topic structure is important. Topics are not the same as raw packets from the car.
|
The code hase been updated with the issues mentioned in the comments above.
|
|
Add detailed MQTT README.md explaining architecture, design choices, and configuration |
| @@ -312,6 +316,17 @@ func (c *Config) ConfigureProducers(airbrakeHandler *airbrake.AirbrakeHandler, l | |||
| producers[telemetry.ZMQ] = zmqProducer | |||
| } | |||
|
|
|||
| if _, ok := requiredDispatchers[telemetry.MQTT]; ok { | |||
| if c.MQTT == nil { | |||
| return nil, errors.New("Expected MQTT to be configured") | |||
There was a problem hiding this comment.
golang errors usually starts with lowercase :-
errors.New("expected MQTT to be configured")There was a problem hiding this comment.
All other related errors in the same method use a capital letter:
"Expected ZMQ/Kinesis/PubSub/Kafka/MQTT to be configured"
Maybe it is better if someone creates a separate pull request to fix all error messages?
(and then maybe also have a look at capital letters used in errors.py)
| SetKeepAlive(time.Duration(config.KeepAlive) * time.Second) | ||
|
|
||
| client := PahoNewClient(opts) | ||
| client.Connect() |
There was a problem hiding this comment.
do we need to do any error handling here?
There was a problem hiding this comment.
ConnectRetry is set. So, the client.Connect() might fail, but that's not a problem because the Paho client will keep trying to connect.
This will make sure a telemetry container can start even if a configured MQTT broker is not online.
Description
This PR add's a MQTT dispatcher. Sending Tesla EV fleet telemetry (e.g., SoC) to an MQTT broker provides real-time, lightweight, and efficient data streaming. MQTT's publish-subscribe model scales well for growing fleets, enabling seamless integration with IoT platforms and data analytics systems for improved monitoring and management.
Type of change
Please select all options that apply to this change:
Checklist:
Confirm you have completed the following steps: