Client is too eager to declare transactions as dead

[abizjak]

Bug Description

The client currently immediately queries the transaction after submitting it to the node. This can cause sporadic failure due to data races especially if multiple nodes are behind a load balancer/proxy.

There is no need to query immediately and adding a delay of 2s to the first query would make the behaviour more predictable.

Alternatively we could have a grace counter for errors. Say 5 failed queries (transaction absent) before we declare it truly dead.

Steps to Reproduce

Put two nodes behind a load balancer that routes each request to a different node.

Expected Result

transaction send for a valid transaction leads to "transaction finalized"

Actual Result

transaction send fails with "Transaction was absent".

┆Issue is synchronized with this Jira Task by Unito