2.6.x: ra_server_proc: send_snapshots/8 runs into an ENOENT

[AVVS]

Describe the bug

At certain point started receiving these under high load:

2024-04-13 19:22:19.609321+00:00 [error] <0.6885.8> Error in process <0.6885.8> on node '[email protected]' with exit value:
2024-04-13 19:22:19.609321+00:00 [error] <0.6885.8> {{badmatch,{error,enoent}},
2024-04-13 19:22:19.609321+00:00 [error] <0.6885.8>  [{ra_server_proc,send_snapshots,8,
2024-04-13 19:22:19.609321+00:00 [error] <0.6885.8>                   [{file,"src/ra_server_proc.erl"},{line,1590}]},
2024-04-13 19:22:19.609321+00:00 [error] <0.6885.8>   {ra_server_proc,'-handle_effect/5-fun-1-',8,
2024-04-13 19:22:19.609321+00:00 [error] <0.6885.8>                   [{file,"src/ra_server_proc.erl"},{line,1257}]}]}

Reproduction steps

Had 10 quorum queues receiving about 70k messages per second (msg is about 64 bytes), quorum was set to 1

Expected behavior

Assume its raft log writer and there is a missing file. It shouldnt crash and should handle the error gracefully

Additional context

No response

[michaelklishin]

RabbitMQ 3.11 is out of community support. Time to upgrade to 3.13.1.

[AVVS]

Planning to do it, however, the bug seems to be in this ra so I was wondering if it was fixed so far? I haven't seen changelog notes that say so, but maybe I missed them. Thought its worth reporting!

[michaelklishin]

RabbitMQ 3.11 ships with Ra 2.6.x which is from 2022. 2.10.0 is almost out, so we won't spend any time investigating exceptions in 2.6.x.

A quick look at ra_server_proc suggests that in main, line 1590 is not a part of the send_snapshots/8 function, and all we can say is that a leader replica has run into an ENOENT response from a file operation when it tried to send snapshots to a newly joint or recovered follower. That can happen for reasons that are completely outside of Ra's control. In other words, the exception itself does not prove the presence of a bug.

[michaelklishin]

More specifically, an enoent was returned when the node tried to read its snapshot state.

#266 seems relevant but it talks about Ra log reinstallation on the follower side, and here the replica is likely the leader (because it tried to send its log state, if the function name is any indication).

I cannot tell much more without a way to reproduce this behavior at least some of the time.

[AVVS]

Thanks for the response. Found it odd as it would just repeatedly do that until I have restarted the node, will update to 3.12.x and hopefully that issue was solved already. 3.13.x is a little fresh to jump to :)

[kjnilsson]

Does it cause a problem?

In artificial tests it is possible that the snapshot sender process tries to send a snapshot that has already been superceded, and thus crashes then tries again.

We could probably handle enoent a bit quieter but we mostly see it during artificial load testin

[AVVS]

This is during production load (consistent hash exchange ->
10 quorum queues, around 70000 messages produces / consumed per second for about 14 hours straight) given the quorum queues with RF 1 (essentially no replication) it doesn't cause problems, but if the RF was higher I suspect in case of failover it could have led to data loss. Not strictly an issue for my usecase, but seems like error should be handled as the file would not magically appear (not sure why it doesnt exist in the first place)

lmk if I can provide more details on the setup that is helpful

[kjnilsson]

Your quorum queues must have at least 3 members for you to see this exception in the logs. The error is not critical and does not affect data safety. There may be a config setting you could tweak that may help. I’ll have a look tomorrow. *Karl Nilsson*

…

On Mon, 15 Apr 2024 at 17:34, Vitaly Aminev ***@***.***> wrote: This is during production load (consistent hash exchange -> 10 quorum queues, around 70000 messages produces / consumed per second for about 14 hours straight) given the quorum queues with RF 1 (essentially no replication) it doesn't cause problems, but if the RF was higher I suspect in case of failover it could have led to data loss. Not strictly an issue for my usecase, but seems like error should be handled as the file would not magically appear (not sure why it doesnt exist in the first place) lmk if I can provide more details on the setup that is helpful — Reply to this email directly, view it on GitHub <#430 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJAHFABRO73DPKRJ6LXYBDY5P6RZAVCNFSM6AAAAABGFWWDYWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMRQG43TG> . You are receiving this because you commented.Message ID: ***@***.***>