Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hermes submits a high number of redundant packets on Stride #3503

Closed
5 tasks
Tracked by #3437
ancazamfir opened this issue Jul 25, 2023 · 5 comments
Closed
5 tasks
Tracked by #3437

Hermes submits a high number of redundant packets on Stride #3503

ancazamfir opened this issue Jul 25, 2023 · 5 comments
Labels
A: bug Admin: something isn't working E: stride External: related to Stride

Comments

@ancazamfir
Copy link
Collaborator

ancazamfir commented Jul 25, 2023

Summary of Bug

[Huge thanks to @romac @ljoss17 and @colin-axner for the help with the investigation]
When relaying on stride the relayers relay an unusual number of redundant ack and recv packets. After our analysis on the transactions, hermes logs and full node logs on thing we found is that stride chain does not emit events for the redundant relays.
When configured with tx_confirmation = true, hermes looks for the events associated with the packet relayed, write_acknowledgement for recv and acknowledge_packet for ack. If it does not find the processing events, it retries a number of times before re-evaluating the original, send_packet for recv and write_acknowledgment for ack.

We confirmed with @colin-axner that in newer ibc-go versions the events for redundant relays are not emitted for redundant relays. He provided these details:

Update
@romac has written tests that show that on ibc-go v4.x:

  • events ARE emitted for redundant acks
  • events ARE NOT emitted for redundant recv packets, so we need to investigate why we do not have a higher number of redundant recv on other chains using v4.

Version

all

Steps to Reproduce

Acceptance Criteria

The number of uneffected packets sent by Hermes on Stride should be comparable with other chains
We should also:

  • if possible add integration test for this case once we understand the root cause and include in "Steps to reproduce"
  • check uneffected packets for chains other than stride on the ibc-go versions mentioned above

For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate milestone (priority) applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@github-project-automation github-project-automation bot moved this to 🩹 Triage in Hermes Jul 25, 2023
@ancazamfir ancazamfir changed the title Hermes does not deal with missing packet events in redundant relays in newer ibc-go versions Hermes does not handle well missing packet events in redundant relays (seen with newer ibc-go versions) Jul 25, 2023
@seanchen1991 seanchen1991 changed the title Hermes does not handle well missing packet events in redundant relays (seen with newer ibc-go versions) Hermes does not handle missing packet events in redundant relays well Jul 25, 2023
@seanchen1991 seanchen1991 moved this from 🩹 Triage to 📥 Todo in Hermes Jul 26, 2023
@seanchen1991 seanchen1991 added A: bug Admin: something isn't working E: stride External: related to Stride labels Jul 26, 2023
@ancazamfir
Copy link
Collaborator Author

An update on this, it seems that at least for Hermes the lack of events does not explain the high number of uneffected packets as we re-evaluate immediately without retrying.
Next steps are to gain access to stride and osmosis full nodes, get funds and run (potentially instrumented) hermes relayer ourselves.
Also in parallel waiting on full node and hermes logs from hermes operators to see if we get some hints on what happens.

@ancazamfir ancazamfir changed the title Hermes does not handle missing packet events in redundant relays well Hermes submits a high number of redundant packets on stride Aug 7, 2023
@seanchen1991 seanchen1991 changed the title Hermes submits a high number of redundant packets on stride Hermes submits a high number of redundant packets on Stride Aug 9, 2023
@ancazamfir
Copy link
Collaborator Author

Probably caused by Stride-Labs/stride#807

@ancazamfir
Copy link
Collaborator Author

Probably caused by Stride-Labs/stride#807

And fixed in Stride-Labs/stride#882

@github-project-automation github-project-automation bot moved this from 📥 Todo to ✅ Done in Hermes Aug 15, 2023
@romac
Copy link
Member

romac commented Aug 16, 2023

Re-opening until we can confirm once Stride v13 rolls out

@romac romac reopened this Aug 16, 2023
@seanchen1991
Copy link
Contributor

@romac Good to close this now? 🙂

@romac romac closed this as completed Aug 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: bug Admin: something isn't working E: stride External: related to Stride
Projects
Status: ✅ Done
Development

No branches or pull requests

3 participants