Messages were not completely delivered from consumer, but successfully ingested into the nats

[chethan-b-hpe]

Recently we had ran scale to test nats but after checking we seen all messages had not been delivered by the consumer and even those messages were not there in the queue ( ex: pushed 5k and from consumer logs we seen around 4000 delivered) , I'm wondering where these messages have been lost, but couldn't able to root cause the issue.

adding to this we are seeing this issue intermittently while consumer.Consume() is calling :

nats: no heartbeat received

checking nats server logs with the above error it was showing this error sometimes:

has NO quorum, stalled

stream configuration : 3 nats server cluster mode

js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
Name: StreamName,
Description: "Webhook Events",
Subjects: []string{
"alpha.>",
nats.InvalidateSubscriptionSubject,
},
MaxBytes: 1024 * 1024 * 1024,
Replicas: 2,
Retention: jetstream.WorkQueuePolicy,
Storage: jetstream.FileStorage,
})

consumer config:

jetstream.ConsumerConfig{ MaxAckPending: 1,
Replicas: 1,
MemoryStorage: true,
MaxDeliver: 21,
BackOff: []time.Duration{
1 * time.Minute, 1 * time.Minute, 1 * time.Minute, 1 * time.Minute, 1 * time.Minute..........}}

[derekcollison]

To eliminate the NATS setup being possibly misconfigured, you can run your app load test against Synadia Cloud. If it behaves as expected then your system is not setup correctly. If it behaves the same, we would want to take a look at your app tier.