2024-10-03 Chain Halt Postmortem

[philipsu522]

Incident Details

This document describes the incident, observations, and root cause analysis for the outage that was experienced on 2024-10-03.

Severity

High (p1)

Impact

• The pacific-1 chain was halted and no blocks were produced.
• All users of pacific-1 were unable to send transactions.
• State was not affected or corrupted.
• No loss of funds.

Duration

1 hour, 37 minutes

Key Metrics

Time	Key Times
2024-10-03 21:37 UTC	Incident Begin Time - Chain halted
2024-10-03 21:43 UTC	Incident Detected Time
2024-10-03 22:30 UTC	Fix Implemented Time
2024-10-03 22:44 UTC	Fix Rollout Start Time
2024-10-03 23:14 UTC	Incident Resolved Time - Chain resumed
2024-10-03 23:42 UTC	Foundation Released Twitter Communications

Incident Timeline

2024-10-03 21:37 UTC (0m)

• At block 106194634, all validators and nodes in pacific-1 chain halted due to a transaction that hit a specific out of gas error.

2024-10-03 21:38 UTC (1m)

• PagerDuty alert NoInServiceRpcNodes fired, and an ecosystem contributor was notified via a Slack notification

2024-10-03 21:52 UTC (15m)

• Engineer got in touch with several technical contributors across the ecosystem and started a war room call to troubleshoot

2024-10-03 22:30 UTC (53m)

• Engineers in the war room committed a potential fix to a branch, which disabled the gas checking in the specific location where the panic occurs.
• Fix was tested and verified

2024-10-03 22:44 UTC (1hr 7m)

• Rollout to all nodes begins
• Validators that weren’t in the war room were notified about the fix

2024-10-03 23:14 UTC (1hr 37m)

• The next block (106194635) reached consensus, as two thirds of voting power upgraded to the hotfix, and block production resumed afterwards.
• Incident Resolved at this time

2024-10-03 23:42 UTC (2hr 5m)

• Foundation releases notification to the community: https://x.com/Sei_FND/status/1841987271051870593

Root Cause

Root Cause Timeline

2024-07-01 - Hooks Pattern Added

• In order to better support EVM interoperability with CW pointer contracts, a new pattern called DeliverTx Hooks that allows for running a special function after the transaction execution completes was added. This served the purpose of adding CW cosmos events to the EVM TX receipt.
  • PR: Translate CW20 events into EVM events in shell receipts #1748
• After analysis, it’s clear that any panic that would happen during these hooks could crash the node.

2024-08-28 - Bug Introduced to Codebase

• A fix was made that would restrict the gas limit for pointer contract wasm queries occurring during the DeliverTx Hook. This adds a panic condition once the gas threshold is crossed.
• There were multiple places where this code was called, and in one of those paths, the panic was not recovered.

2024-09-16 - Bug Introduced to Pacific-1

• v5.8.0 was released and pacific-1 became exposed to this introduced bug.
• Block height was 102491599 for the upgrade.

2024-10-03 - Panic Triggered

• Block 106194634 containing suspected transaction 0xd3c08dea4be3915779925d0e933b6cf5b5c9cbbab51974fee66bfd5a2383f73a (explorer link) causes a panic on all validator and RPC nodes, causing a chain halt.

Resolution

1. Refactor hooks to be more panic tolerant: Refactor DeliverTx hook so that panics can be handled sei-cosmos#543
2. Add static analysis to code base to detect panics
3. Add transactions with multiple wasm events that could run out of gas to unit tests
4. Create replay tool to replay transactions to prevent future regression