json from 500 errors from rpcs is parsed

[novalis]

Describe the bug
500 errors from tezos node rpcs are not actually retried:

tezos-reward-distributor/src/rpc/rpc_reward_api.py

Line 406 in d718062

# server side errors 5xx can pass for a retry

This says "server side errors 5xx can pass for a retry". But a retry is not what happens -- the body of the 5xx response is returned.

To Reproduce

I discovered this while working on #596 -- but you could just point tezos_node_addr to an endpoint that returns 500 for queries to e.g. /chains/main/blocks/\d+/context/selected_snapshot?cycle=\d+.

Expected behavior
I would expect these queries to either cleanly fail, or to get retried (and then cleanly fail). But instead, the 500 response is parsed as json (which, in the case of errors from tezos, it is), and returned to the caller, causing an unclean error later on.

[novalis]

When was this completed? I don't see it in !581.

[jdsika]

Did I misinterpret something? I thought you solved it in #596 and it was merged into #581 by Nicolas and I just merged it into master?

[jdsika]

ahhhh, you discovered it in #596

sorry, reopening

[rvermootenct]

Hey there @novalis, I tried to reproduce this issue, but may be missing something here. Is it possible you were getting a 4xx error? If I ensure it's a 5xx error it indeed does retry. Can you confirm for me?

[novalis]

I am no longer working on Tezos, so I am not set up to reproduce. But the code appears the same: 500 errors are not caught by the if statement, and are in fact parsed as JSON.