dynamic subject mapping/partitioning in jetstream

[yuliya-tenovos]

we have multiple customers publishing messages to the stream. At times one or two customers will generate a lot of messages. I would like to be able to add or remove consumers based on the load. At the same time if one customer is generating a lot of messages , those messages should be handled in order. My understanding of the subject mapping or partitioning that its static. If I create 10 partitions, I will have to run 10 consumers listening to each partition? what I actually want is this: if two customers generate many messages at the moment: handle them by two separate consumers, let the other 5 customers be processed by the third one. If the number of messages in the stream decreases , I want to be able to scale back to just one consumer

[jnmoyne]

You will be able to do basically what you want (distribute the consumption of messages between consumers while still retaining the strict ordering of the messages subject, and be able to scale up and down the number of consumers administratively) with a new 'consumer groups' client-side library that we will be able to release after NATS server version 2.11 is released, and which indeed relies on the deterministic partitioning functionality of the subject transformation feature. At a high level, the technique is to use the 'maximum number of consumers' as your number of partitions and then have the actual consumers currently deployed distribute themselves over those numbers of partitions: e.g. if you have 10 partitions: with one consumer, it will consume from all 10 partitions, with two consumers they will each consume from 5 partitions, etc... up to a maximum of 10 consumers.

[yuliya-tenovos]

Thank you! that sounds promising. Do we know approximately when this will be released?
One other question, if I have 10 partitions and 20 customers, for example. Does it mean that customer always goes to exactly same partition? Even if there were no messages coming out for long period of time? Or is it only for the messages currently in the stream. I am a little confused about it. so if customer 1, 2 are assigned to the same partition, and customer 1 generated a 10k of messages. the customer 2 is stuck being in the same queue ? I hope the question makes sense

[niondir]

@jnmoyne any update on the new 'consumer groups' client-side library?

I can't find anything about it yet. Is there an open issue regarding the state of implementation?