Leaf-node JetStream retention policy for hub-sourced streams

[AlexeiSosnov]

It is not very clear from documentation how retention works for leaf-node JetStreams.

We have a leaf-node with JetStream S1 used for reliably replicate messages to a hub-node JetStream SS1, where the latter is sourced from the former leaf-node JetStream S1. However, once a message is replicated from the leaf-node JS S1 to to the hub-node JS SS1, we do not need to store the message on the leaf node (whether in memory or in a file). So the questions:

• is the hub-node works as a consumer for the leaf-node stream?
• if yes, what is the best retention policy for the leaf-node stream (will WorkQueue retention policy is ok)?
• if the hub-node is not a consumer for sourced stream on the leaf-node, how to clear leaf-node stream messages once they are delivered/replicated to the hub-node?

[derekcollison]

The consumer connection that exists between sources and mirrors is ephemeral, and so could lead to inconsistent results if the leafnode stream would be interest based policy.

We do have a few folks who use WQ semantics, but there have been reports during long network outages of messages not being properly cleaned up.

We plan to do some more work here for 2.11 and beyond. For now I would recommend using age limits for the stream on the leafnode that encompasses the longest outage you could expect for a network event. Note you can use multiple limits, so setting max bytes is recommended as well such that the leafnode stream does not run away with disk and or memory usage.

[AlexeiSosnov]

Derek,

Thank your for your answer. We planned to use NATS for some kind of streaming over "mostly reliable" connection, using NATS ready-provided limits and flow-control features (in addition to many other "transport species" NATS provides). So the idea is to use

• WQ semantics
• in addition with age and size limits on the leaf-node stream and DiscardNew policy (according to docs, limits are additive to other policies)

Should it work?

To my understanding, interest-based policy can lead to lost messages, if these messages are published when leaf-node upstream connection is not yet established or even when upstream connection outage, due to ephemeral consumers are automatically deleted when inactive. Is it true?

[derekcollison]

Regrading interest policy yes that is correct.

I would use limits policy if you need to make sure it gets to the hub side.

[AlexeiSosnov]

Thanks.
If I would use only limits policy, stream storage congestion is possible, if I transfer data more frequently than maximum possible network outage and normally connection works. So, will WQ policy + age limit provide both fast retention when traffic flows normally as well as fallback retention when outage is too long?

[derekcollison]

The age limit should be your maximum outage window allowance.