MQTT clients aren't behaving as expected using the Confluent Cloud IoT Ingest and Control solution for Zilla

[vordimous]

We removed TLS on the MQTT side to make it simpler.
Now we are able to connect with MQTT Explorer to the instance and publish a message. We see that data is written to the Kafka topics session, messages and retain.

But it is not working correctly

• Only very few messages are actually written to the Kafka Topics
• Subscribing & Receiving of the telegrams does not work at all.
• Sometimes the MQTT client disconnects and is not able to reconnect for a long time.

Confluent Cloud IoT Ingest and Control solution docs

logs:

public.mqtt_server [08/Aug/2024:12:10:52 +0000] BINDING_MQTT_CLIENT_CONNECTED Session authorization (-) was successful for client id (mqtt-explorer-a325789).

zilla.yaml:

name: public
telemetry:
  metrics:
    - stream.opens.sent
    - stream.closes.sent
    - stream.errors.sent
    - stream.errors.received
    - stream.closes.received
    - stream.opens.received
  exporters:
    stdout:
      type: stdout
bindings:
  tcp_server:
    type: tcp
    kind: server
    telemetry:
      metrics:
        - stream.*
    options:
      host: 0.0.0.0
      port: 1883
    exit: mqtt_server
  mqtt_server:
    type: mqtt
    kind: server
    telemetry:
      metrics:
        - stream.*
    options:
      versions:
        - v5
        - v3.1.1
    exit: mqtt_kafka_mapping
  mqtt_kafka_mapping:
    type: mqtt-kafka
    kind: proxy
    telemetry:
      metrics:
        - stream.*
    options:
      topics:
        sessions: iot-zilla-mqtt-sessions
        messages: iot-zilla-mqtt-messages
        retained: iot-zilla-mqtt-retained
      clients:
        - spBv1.0/PAULv0/+/{identity}/#
    exit: kafka_cache_client
  kafka_cache_client:
    type: kafka
    kind: cache_client
    telemetry:
      metrics:
        - stream.*
    exit: kafka_cache_server
  kafka_cache_server:
    type: kafka
    kind: cache_server
    telemetry:
      metrics:
        - stream.*
    options:
      bootstrap:
        - iot-zilla-mqtt-messages
        - iot-zilla-mqtt-retained
    exit: kafka_client
  kafka_client:
    type: kafka
    kind: client
    options:
      servers:
        - ${{aws.secrets.arn:aws:secretsmanager:*:secret:*#bootstrap_server}}
      sasl:
        mechanism: plain
        username: "${{aws.secrets.arn:aws:secretsmanager:*:secret:*#api_key}}"
        password: "${{aws.secrets.arn:aws:secretsmanager:*:secret:*#api_secret}}"
    telemetry:
      metrics:
        - stream.*
    exit: tls_client
  tls_client:
    type: tls
    kind: client
    vault: secure
    telemetry:
      metrics:
        - stream.*
    exit: tcp_client
  tcp_client:
    type: tcp
    kind: client
    telemetry:
      metrics:
        - stream.*

[bmaidics]

Just to make sure: when you created the topics for the sessions (iot-zilla-mqtt-sessions), did you created it as compact? Also the retained topic (iot-zilla-mqtt-retained) should be compact.

[bmaidics]

Also I can see based on your config you have clients mqtt-kafka option: spBv1.0/PAULv0/+/{identity}/#
The client-id that I see on the logs doesn't match this. Can you try to remove that option from the config and see if this resolves the issue?

[yveskurz]

These are the topic configuration on confluent cloud

Further, I've set the client id to the topic I'm posting message to. So the clientID is P01010151iae4N5, topic I'm publishing to is spBv1.0/PAULv0/DDATA/P01010151iae4N5/P12344

It sometimes, but very rarely, the message get's produced to the messages topic, but it takes a very long time that the mqtt message appears on the Kafka topic (like 10s of seconds).
In the attached Wireshark capture, MQTT Explorer connected to zilla, tried to subscribe to three topics, then I sent three publish telegrams, none of them appeared on the messages topic. Also the MQTT client disconnects from zilla for some unknown reason and is no longer able to establish connection.
mqtt_explorer_zilla.pcapng.zip

Also as @bmaidics suggested, I've removed the clients option from mqtt-kafka completely and restarted zilla. The result was exactly the same.

[bmaidics]

Thanks for the informations @yveskurz . I managed to reproduce the issue using my local setup with MQTT Explorer.
The issue only reproducible using that client (we haven't tested that client yet), if you want to be unblocked quickly while we're investigating the issue please use other client (MQTTX, postman, mosquitto client). We'll update you once we know the root cause.

[jfallows]

@yveskurz we have found and fixed the issue reported with MQTT Explorer and verified Confluent Cloud connectivity as well with the latest Zilla release.

#1189

Note: An updated Zilla Plus release is in flight including the fix.