configure offline publishing in v2

[shravan097]

Is your feature request related to a problem? Please describe.
I did not see if offline publishing was enabled by default.

Describe the solution you'd like
As in python sdk v1, it would be nice to see something like configureOfflinePublishQueueing method.

Describe alternatives you've considered
I would have a Queue of my own maintaining if the message was published or not.

Additional context
https://github.com/aws/aws-iot-device-sdk-python/blob/master/samples/basicPubSub/basicPubSub.py#L101

[graebm]

The current behavior is:

• All published messages are queued before sending, regardless of whether the client is offline or online.
• When the client gets online (or if it is already online) the messages from the queue are sent in order.
• HOWEVER: If the client WAS online, and goes offline, at that moment it cancels every PUBLISH and SUBSCRIBE in its queue.

We understand this is inconsistent. The user probably wants either:

1. messages are always queued until they can be sent
2. messages fail if the client is offline, or goes offline before they are sent.

We are very actively evaluating changing this. I will keep this issue open and we'll update when we've made changes (or formalized the current behavior)

If you want to take full control of your destiny right now, you could build a queue of your own that sends 1 message at a time to the client. If it succeeds, then send the next. If it fails, then retry. This also gives you the opportunity to have full control over the length of time a message can be queued, the number of messages that will be queued, etc.

[samvandamme]

If you want to take full control of your destiny right now, you could build a queue of your own that sends 1 message at a time to the client. If it succeeds, then send the next. If it fails, then retry. This also gives you the opportunity to have full control over the length of time a message can be queued, the number of messages that will be queued, etc.

Hi @graebm. How would you go about that? If my device publishes messages, but it's offline, I would think those messages stay in soms sort of queue? Is there a way to access this queue and purge certain messages from it if they have not been published yet?

Thank you

[jmklix]

@samvandamme you can set quality of service (qos) to at most once (0).
qos=mqtt.QoS.AT_MOST_ONCE
On the same device subscribe to the topic that you are publishing to. Then to make sure messages are sent you maintain a list of messages that you have published. Only remove them when you have received back the message on the topic.

[dg-eparizzi]

The current behavior is:

• All published messages are queued before sending, regardless of whether the client is offline or online.
• When the client gets online (or if it is already online) the messages from the queue are sent in order.
• HOWEVER: If the client WAS online, and goes offline, at that moment it cancels every PUBLISH and SUBSCRIBE in its queue.

We understand this is inconsistent. The user probably wants either:

1. messages are always queued until they can be sent
2. messages fail if the client is offline, or goes offline before they are sent.

We are very actively evaluating changing this. I will keep this issue open and we'll update when we've made changes (or formalized the current behavior)

If you want to take full control of your destiny right now, you could build a queue of your own that sends 1 message at a time to the client. If it succeeds, then send the next. If it fails, then retry. This also gives you the opportunity to have full control over the length of time a message can be queued, the number of messages that will be queued, etc.

So what's the recommended approach to handle devices going offline for some periods of time?

Assuming I have a device sending messages every 5 seconds, the device is online, messages are reaching AWS IoT Core. Then, the device lost connection for 10 minutes and then goes online again. What happens with all those messages that were tried to publish during the 10 minutes offline period? Do they get published as soon as the device goes online again or are they lost for ever?

[jmklix]

The first few messages will be lost while the sdk hasn't realized that it is no longer disconnected. All subsequent messages would be save in the queue and sent when connection is restored.

When testing this I lost the first 5 messages when using the pubsub, which sends messages every second. So you would only loose one message most of the time.

[nicholashughes]

I'd also like to understand this in greater detail. As mentioned above, messages that are published while offline go into the queue and then are published when the connection goes back online.

How many messages are stored in the offline queue? The previous SDK allowed you to choose, infinite, none, or a specified amount. I can't find any documentation about this. It sounds like infinite but I'd like to know. The previous SDK also allowed you to drop from the top or bottom of the queue. Is that possible now?

The previous SDK also allowed you to choose a drain rate for messages stored in the queue. Is this configurable now? If not, what is the value?

[Blake-James]

@graebm Are you able to let us know if there is anything currently on the roadmap for this?
Should we expect any sort of controls over the offline queue in addition to the default behaviour described here?

[Infinity1high]

Agree, previous SDK allowed more, that is weird to build smth that is missing features and claiming this solution is better

[Divya-nemuri]

i want to know the connection status of the client, while publishing the data if there is no internet connection i wanna store data in the local, i was trying to get the status. i tried with publish_future,data = MQTT_CONNECTION.publish()
was checking publish_future.result() if data is published it is returning packet id but if i disconnect the internet connection it is just stopping the execution there. i tried reading the future.result() of MQTT_CONNECTION which is always "session_present":True.

is there any way to get the connection status or the message published status.

[Divya-nemuri]

The current behavior is:

• All published messages are queued before sending, regardless of whether the client is offline or online.
• When the client gets online (or if it is already online) the messages from the queue are sent in order.
• HOWEVER: If the client WAS online, and goes offline, at that moment it cancels every PUBLISH and SUBSCRIBE in its queue.

We understand this is inconsistent. The user probably wants either:

1. messages are always queued until they can be sent
2. messages fail if the client is offline, or goes offline before they are sent.

We are very actively evaluating changing this. I will keep this issue open and we'll update when we've made changes (or formalized the current behavior)

If you want to take full control of your destiny right now, you could build a queue of your own that sends 1 message at a time to the client. If it succeeds, then send the next. If it fails, then retry. This also gives you the opportunity to have full control over the length of time a message can be queued, the number of messages that will be queued, etc.

How to check if the message sent to the client is succeeded or not

[jmklix]

@Divya-nemuri you can have the client publish a message indicating that it has received a message

Also for connection status:

If you want another device/program to know if a device is online you will want to use Lifecycle events.

If you want a device to know when it is online/online you will want to use on_connection_interrupted.

[juvi87]

has there been any progress on this?

[jmklix]

With mqtt5 you can now configure the offline_queue_behavior. Check our API docs here for more info, but here is how it currently works:

class awscrt.mqtt5.ClientOperationQueueBehaviorType(value)

Controls how disconnects affect the queued and in-progress operations tracked by the client. Also controls how operations are handled while the client is not connected. In particular, if the client is not connected, then any operation that would be failed on disconnect (according to these rules) will be rejected.

DEFAULT = 0
Default client operation queue behavior. Maps to FAIL_QOS0_PUBLISH_ON_DISCONNECT.

FAIL_NON_QOS1_PUBLISH_ON_DISCONNECT = 1
Re-queues QoS 1+ publishes on disconnect; un-acked publishes go to the front while unprocessed publishes stay in place. All other operations (QoS 0 publishes, subscribe, unsubscribe) are failed.

FAIL_QOS0_PUBLISH_ON_DISCONNECT = 2
QoS 0 publishes that are not complete at the time of disconnection are failed. Un-acked QoS 1+ publishes are re-queued at the head of the line for immediate retransmission on a session resumption. All other operations are requeued in original order behind any retransmissions.

FAIL_ALL_ON_DISCONNECT = 3
All operations that are not complete at the time of disconnection are failed, except operations that the MQTT5 spec requires to be retransmitted (un-acked QoS1+ publishes).

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.