RabbitMQ High Memory Usage of Classic Mirror Queue #1788

kunlunshanmonster · 2024-12-13T09:16:52Z

kunlunshanmonster
Dec 13, 2024

my rabbitmq, named rabbitmq01（3.9.17） deployed on Kubernetes 1.19, with resource configuration of 8C16G7pod. the queue is a durable classic mirror queue with 4 replicas, the message is persistent .When I use high rate（980message/s） to send messages and low rate to consume messages （20message/s）, over time, The broker memory keeps increasing, even exceeding the high memory watermark, memory configuration is: vm_memory_high_watermark_paging_ratio = 0.5，vm_memory_high_watermark.relative = 0.4. when memory exceed high memory watermark,I stop to produce and consume message, the memory no longer changes and there are 1 million messages in the queue， I produce memory use breakdown using Management UI, its shows:

 Classic Queues: 1.4GiB queues, 7.9MiB mirrors 
Binaries: 2.9GiB
 Preallocated Memory: 258MiB preallocated by runtime, unused. 1.4GiB unallocated, reserved by the OS 
Runtime Used: 4.4GiB
 Runtime Allocated: 4.6GiB 
Resident Set Size(RSS) reported by the OS: 6.1GiB

My question is that I have already configured vm_memory_high_watermark_paging_ratio and vm_memory_high_watermark.relativ. Therefore, when the memory usage exceeds 20% of the total memory, the message should paged from memory to disk to free up memory space. The memory usage should not exceed the high level, but in reality, it exceeds the high level.

In addition, I use another MQ cluster (named rabbitmq02)with the same configuration to send and consume messages at the same rate（send rate is 980/s, consume rate is 20/s）. The memory usage in this MQ cluster is around 2GiB, and there are 5 million messages in the queue. I think this phenomenon is normal， so why did the first rabbitmq(rabbitmq01) broker's memory use exceed high memory watermark？