feat(indexer/postgres)!: add basic support for header, txs and events

[facundomedica]

Description

Related to epic #20352

• Added failure message if the indexer fails on block 1, as we can't rollback from that, the user will have to reset the node before restarting, otherwise the indexer will have missed that block.
• Added txDecoder to baseapp listenerWrapper, which will then be used to get the JSON of the decoded tx.
• Added BlockNumber to event and TxData
• Added some fixes for v2

Missing:

• tx hash. For this we need to use the decoded tx's Hash method, which is not accessible from the postgres indexer side, and if we want to access it from the baseapp side then we'd need to decode it before.

---

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

• included the correct type prefix in the PR title, you can find examples of the prefixes below:
• confirmed ! in the type prefix if API or client breaking change
• targeted the correct branch (see PR Targeting)
• provided a link to the relevant issue or specification
• reviewed "Files changed" and left comments if necessary
• included the necessary unit and integration tests
• added a changelog entry to CHANGELOG.md
• updated the relevant documentation or specification, including comments for documenting Go code
• confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

Please see Pull Request Reviewer section in the contributing guide for more information on how to review a pull request.

I have...

• confirmed the correct type prefix in the PR title
• confirmed all author checklist items have been addressed
• reviewed state machine logic, API design and naming, documentation is accurate, tests and test coverage

Summary by CodeRabbit

• New Features

  • Enhanced error handling during commit operations to prevent invalid rollbacks.
  • Added support for block height information in transaction and event processing.
  • Introduced new methods for handling transactions and events in the listener.
  • Enhanced event handling with block-specific information for better traceability.
  • Added a new method for module codec management in the accounts module.

• Bug Fixes

  • Improved error management in transaction decoding, allowing block finalization despite individual transaction failures.

• Documentation

  • Updated module dependencies and replace directives for better local development support.

• Chores

  • Added new dependencies for PostgreSQL support and updated existing ones for various modules.

[coderabbitai]

📝 Walkthrough

📝 Walkthrough

Walkthrough

The changes in this pull request involve multiple modifications across various files, primarily focusing on enhancing error handling, logging, and event processing capabilities. Key updates include improvements to the Commit method in the BaseApp for better error management at block height 1, enhancements to transaction and event handling in the streaming and indexer components, and updates to the handling of protobuf message types in the codec. Additionally, several go.mod files have been updated to manage dependencies more effectively, particularly with local paths for the cosmossdk.io/schema module.

Changes

File	Change Summary
baseapp/abci.go	Modified Commit method to handle errors at block height 1, enhanced logging for commit errors.
baseapp/streaming.go	Updated listenerWrapper to include txDecoder, modified EnableIndexer, and enhanced event processing with block height information.
codec/collections.go	Updated protoCol function to handle BytesKind fields as nullable and improved schema field handling for google.protobuf.Timestamp and google.protobuf.Duration.
go.mod	Added new dependency cosmossdk.io/schema, modified replace directives for local paths.
indexer/postgres/base_sql.go	Altered SQL schema for tx and event tables, allowing null values and adding new columns.
indexer/postgres/listener.go	Introduced OnTx and OnEvent methods for processing transaction and event data, including error handling.
runtime/v2/app.go	Enhanced SchemaDecoderResolver to handle OverrideStoreKeys from the App config.
runtime/v2/go.mod	Added replace directive for cosmossdk.io/schema pointing to a local path.
schema/appdata/data.go	Added BlockNumber field to TxData and Event structs.
schema/decoding/middleware.go	Modified error handling in Middleware function to skip missing codecs instead of terminating.
server/v2/cometbft/abci.go	Updated transaction handling in FinalizeBlock and internalFinalizeBlock methods for improved error logging.
server/v2/cometbft/go.mod	Added replace directive for cosmossdk.io/schema pointing to a local path.
server/v2/cometbft/streaming.go	Modified streamDeliverBlockChanges method to accept decoded transactions and updated listener callbacks to include block height.
server/v2/go.mod	Added replace directive for cosmossdk.io/schema and a new require statement for its version.
server/v2/stf/go.mod	Added replace directive for cosmossdk.io/schema pointing to a local path.
server/v2/stf/stf.go	Set BlockNumber for events in deliverTx, preBlock, beginBlock, and endBlock methods.
simapp/go.mod	Updated dependencies and added new replace directives for local paths.
simapp/v2/app_di.go	Added imports for PostgreSQL driver and indexer, indicating potential future functionality.
simapp/v2/go.mod	Added new dependencies for PostgreSQL support and updated replace directives.
tests/go.mod	Updated to include cosmossdk.io/schema with a replace directive and modified dependencies.
x/accounts/go.mod	Updated dependencies and added new indirect dependencies with a replace directive for cosmossdk.io/schema.
x/accounts/keeper.go	Enhanced NewKeeper function to improve clarity of collections.Map instances with named keys.
x/accounts/module.go	Added ModuleCodec method to AppModule for improved codec management.
x/distribution/go.mod	Updated dependencies and added a replace directive for cosmossdk.io/schema.

Possibly related PRs

• feat(schema): update event index definitions #21565: Updates the Event struct in schema/appdata/data.go, related to event handling improvements.
• feat(store): migrate db backends to store/db and remove the cosmos-db deps completely #21765: Enhances event handling in the indexer and streaming components, aligning with error handling improvements.
• feat(server/v2/cometbft,stf): Listener integration in server/v2 #21917: Integrates appdata.Listener into the server, related to event handling capabilities.
• feat(baseapp): integrate the appdata.Listener in baseapp #21965: Integrates appdata.Listener in the BaseApp, improving event tracking during block processing.
• feat(server/v2): init the indexer in server/v2 #22218: Initializes the indexer in the server, related to overall architecture enhancements.
• build(deps): bump x/tx #22327: Updates the x/tx dependency, relevant to transaction handling logic.
• build(deps): use cosmossdk.io/core v1.0.0-alpha.5 #22369: Updates the cosmossdk.io/core dependency, affecting the Commit method functionality.
• fix(simapp/v2): full AutoCLI support #22410: Addresses AutoCLI support, indirectly relating to the Commit method changes.
• chore: prepare core changelog #22495: Prepares the changelog for core changes, documenting improvements made in the Commit method.
• fix(testutil/integration): use only one context in integration test framework #22616: Simplifies context management in the integration test framework, related to error handling improvements.

Suggested labels

C:x/staking, C:CLI, C:x/gov, C:x/slashing, C:x/mint, C:x/bank, C:x/params, C:x/evidence, C:x/upgrade, C:x/feegrant, C:x/authz, C:x/group, C:x/nft, C:x/consensus, C:x/circuit, C:x/protocolpool, C:x/epochs, C:x/accounts/lockup, C:x/accounts/multisig, C:x/accounts/base, C:orm

Suggested reviewers

• kocubinski
• testinginprod
• sontrinh16
• julienrbrt
• aaronc

---

📜 Recent review details

Configuration used: .coderabbit.yml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 9ee52c6 and a7dabcc.

⛔ Files ignored due to path filters (7)

• runtime/v2/go.sum is excluded by !**/*.sum
• server/v2/cometbft/go.sum is excluded by !**/*.sum
• server/v2/go.sum is excluded by !**/*.sum
• simapp/go.sum is excluded by !**/*.sum
• tests/go.sum is excluded by !**/*.sum
• x/accounts/go.sum is excluded by !**/*.sum
• x/distribution/go.sum is excluded by !**/*.sum

📒 Files selected for processing (5)

• runtime/v2/go.mod (1 hunks)
• server/v2/cometbft/go.mod (1 hunks)
• simapp/go.mod (1 hunks)
• tests/go.mod (1 hunks)
• x/accounts/go.mod (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (5)

• server/v2/cometbft/go.mod
• tests/go.mod
• runtime/v2/go.mod
• simapp/go.mod
• x/accounts/go.mod

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Generate unit testing code for this file.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (2)

indexer/postgres/listener.go (2)

90-115: Add function documentation and improve error messages

The transaction listener implementation would benefit from:

1. GoDoc comments explaining the function's purpose and parameters
2. More descriptive error messages that include context about the transaction

Apply this diff to add documentation:

+// txListener returns a function that processes and stores transaction data in PostgreSQL.
+// It handles both raw transaction bytes and JSON representations.
 func txListener(i *indexerImpl) func(data appdata.TxData) error {

Also, enhance error messages:

-				return err
+				return fmt.Errorf("failed to get transaction bytes at height %d, index %d: %w", 
+					td.BlockNumber, td.TxIndex, err)

---

145-149: Enhance error context in SQL operations

The error message could provide more context about the event being processed.

Apply this diff to improve error handling:

 			_, err := i.tx.Exec("INSERT INTO event (block_number, block_stage, tx_index, msg_index, event_index, type, data) VALUES ($1, $2, $3, $4, $5, $6, $7)",
 				e.BlockNumber, e.BlockStage, e.TxIndex, e.MsgIndex, e.EventIndex, e.Type, jsonData)
 			if err != nil {
-				return fmt.Errorf("failed to index event: %w", err)
+				return fmt.Errorf("failed to index event (block: %d, type: %s, tx_index: %d): %w",
+					e.BlockNumber, e.Type, e.TxIndex, err)
 			}

📜 Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 689c530 and 9a575f0.

📒 Files selected for processing (2)

• indexer/postgres/base_sql.go (1 hunks)
• indexer/postgres/listener.go (2 hunks)

🚧 Files skipped from review as they are similar to previous changes (1)

• indexer/postgres/base_sql.go

🧰 Additional context used

📓 Path-based instructions (1)

indexer/postgres/listener.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.

🔇 Additional comments (3)

indexer/postgres/listener.go (3)

4-4: LGTM!

The added json package import is necessary for the new JSON handling functionality.

---

85-88: Consider splitting the code into separate files

As suggested in a previous review, consider breaking out the transaction and event handling code into separate files (tx.go and event.go) to improve code organization and maintainability.

---

134-143: Preserve all event attributes by handling duplicate keys appropriately

The current implementation may lose data when events have multiple attributes with the same key.

[julienrbrt]

Adding backport label for SDK changes. The modules tagged from main can be deleted from the backport.

[julienrbrt]

utACK

[coderabbitai]

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (1)

baseapp/abci.go (1)

1002-1005: LGTM! Consider enhancing the error message.

The special handling for block height 1 failures is a good addition that prevents invalid rollbacks and provides clear guidance.

Consider enhancing the error message to provide more guidance:

-						return nil, fmt.Errorf("failed to commit block 1, can't automatically rollback: %w", err)
+						return nil, fmt.Errorf("failed to commit block 1, manual node reset required (can't automatically rollback to height 0): %w", err)

📜 Review details

Configuration used: .coderabbit.yml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 9a575f0 and 9ee52c6.

⛔ Files ignored due to path filters (2)

• go.sum is excluded by !**/*.sum
• simapp/v2/go.sum is excluded by !**/*.sum

📒 Files selected for processing (12)

• baseapp/abci.go (1 hunks)
• go.mod (1 hunks)
• runtime/v2/go.mod (1 hunks)
• server/v2/cometbft/abci.go (3 hunks)
• server/v2/cometbft/go.mod (1 hunks)
• server/v2/go.mod (1 hunks)
• simapp/go.mod (1 hunks)
• simapp/v2/go.mod (3 hunks)
• tests/go.mod (1 hunks)
• x/accounts/go.mod (1 hunks)
• x/accounts/module.go (2 hunks)
• x/distribution/go.mod (1 hunks)

🚧 Files skipped from review as they are similar to previous changes (11)

• server/v2/cometbft/go.mod
• x/accounts/module.go
• server/v2/go.mod
• runtime/v2/go.mod
• go.mod
• tests/go.mod
• x/accounts/go.mod
• x/distribution/go.mod
• server/v2/cometbft/abci.go
• simapp/go.mod
• simapp/v2/go.mod

🧰 Additional context used

📓 Path-based instructions (1)

baseapp/abci.go (1)

Pattern **/*.go: Review the Golang code for conformity with the Uber Golang style guide, highlighting any deviations.