-
Notifications
You must be signed in to change notification settings - Fork 33
Persistence State Hash Protocol
This document describes the Persistence
module's internal implementation of how the state hash is computed. Specifically, it defines the 'Compute State Hash' flow in the shared architectural state hash flow defined here.
Alternative implementation of the persistence module are free to choose their own State Storage engines (SQL, KV stores, etc) or their own State Commitment paradigms (Merkle Trees, Vector Commitments, etc), but the output hash must remain identical.
The state hash is a single 256 bit digest that takes a snapshot of the world state at any committed height. It is needed to guarantee and prove the integrity of the world state, and is what's referenced in every block header when building any blockchain.
This document defines how Pocket V1 takes a snapshot of its world state. An introduction to the requirements, types and uses of hashes in blockchain systems is outside the scope of this document.
Component | Data Type | Implementation Options - Examples | Implementation Selected - Current | Example | Use Case |
---|---|---|---|---|---|
Data Tables | SQL Database / Engine | MySQL, SQLite, PostgreSQL | PostgresSQL | Validator SQL Table | Validating & updating information when applying a transaction |
Merkle Trees | Merkle Trie backed by Key-Value Store | Celestia's SMT, Libra's JMT, Cosmos' IAVL, Verkle Tree | Pocket's SMT (Forked from Celestia) | Fisherman Trie | Maintains the state of all account based trees |
Blocks | Serialization Codec | Amino, Protobuf, Thrift, Avro | Protobuf | Block protobuf | Serialized and inserted into the Block Store |
Objects (e.g. Actors) | Serialization Codec | Amino, Protobuf, Thrift, Avro | Protobuf | Servicer protobuf | Serialized and inserted into the corresponding Tree |
Block Store | Key Value Store | LevelDB, BadgerDB, RocksDB, BoltDB | BadgerDb | Block Store | Maintains a key-value store of the blockchain blocks |
Transaction Indexer | Key Value Store | LevelDB, BadgerDB, RocksDB, BoltDB | BadgerDB | Tx Indexer | Indexes transactions in different ways for fast queries, presence checks, etc... |
The block protobuf that is serialized and store in the block store can be found in persistence/proto/block_persistence.proto
. This proto contains the stateHash
along with the corresponding height.
An individual Merkle Tree is created for each type of actor, record or data type. Each of these is backed by its own key-value store.
Actor Merkle Trees:
- Applications
- Validators
- Fisherman
- Servicers
Account Merkle Trees:
- Accounts
- Pools
Data Merkle Trees
- Transactions
- Parameters
- Flags
Note: GetRecordsUpdatedAtHeight
is an abstraction for retrieving all the records from the corresponding SQL tables depending on the type of record (Actors, Transactions, Params, etc...)
This flow shows the interaction between the PostgresDB
and MerkleTrees
listed above to compute the state hash. Assuming the process of applying a proposal block to the current context (i.e. the uncommitted SQL state) is done, the following steps compute the hash of the new world state.
- Loop over all of the merkle tree types
- Use
GetRecordsUpdatedAtHeight
to retrieve all the records updated at the context's height - Serialize each record using the corresponding underlying protobuf
- Insert the serialized record into the corresponding tree (which is back by a key-value store)
- Compute the root hash of each tree
- Insert the name of the tree and its root hash into the root tree
- Compute the new
stateHash
by hex encoding the root tree's root hash
sequenceDiagram
participant P as Persistence
participant PSQL as Persistence (SQL Store)
participant PKV as Persistence (Key-Value Store)
loop for each merkle tree type
P->>+PSQL: GetRecordsUpdatedAtHeight(height, recordType)
PSQL->>-P: records
loop for each state tree
P->>+PKV: Update(addr, serialize(record))
PKV->>-P: result, err_code
end
P->>+PKV: GetRoot()
PKV->>-P: rootHash
P->>P: rootTree.Update(stateTreeName, rootHash)
end
P->>P: stateHash = hex(rootTree.Root())
activate P
deactivate P
When the new state hash is computed, the different state trees read the updates from their respective Postgres tables and update the trees accordingly.
IBCMessage
objects are inserted into the ibc_entries
table in two ways., depending on the IBC messages' type: 1. UpdateIBCStore
: the key
and value
fields are inserted with the height into the table 2. PruneIBCStore
: the key
with a nil
value is inserted into the table
For each entry in the ibc_message
table depending on the entries value
field the tree will perform one of two operations:
-
value == nil
- This is a
PruneIBCStore
message and thus the tree will delete the entry with the givenkey
ibcTree.Delete(key)
- This is a
-
value != nil
- This is an
UpdateIBCStore
message and thus the tree will update the entry with the givenkey
to have the givenvalue
ibcTree.Update(key, value)
- This is an
Note: Prior to insertion the key
and value
fields of the messages are hexadecimally encoded into strings.
When the Commit(proposer, quorumCert)
function is invoked, the current context is committed to disk. The PersistenceContext
does the following:
- Read data from the persistence context's in-memory state
- Prepare a instance of the
Block
proto & serialize it - Insert the
Block
into theBlockStore
- Insert the
Block
into the SQL Store - Commit the context's SQL transaction to disk
sequenceDiagram
participant P as Persistence
participant PSQL as Persistence (SQL Store)
participant PKV as Persistence (Key-Value Store)
P->>P: prepare & serialize block proto
activate P
deactivate P
%% Insert into the SQL store
P->>+PSQL: Insert(height, block)
PSQL->>-P: result, err_code
%% Insert into the Block Store (i.e. Key-Value store)
P->>+PKV: Put(height, block)
PKV->>-P: result, err_code
%% Commit the SQL transaction
P->>+PSQL: Commit(SQL Tx to disk)
PSQL->>-P: result, err_code
TODO: If an error occurs at any step, all of the operations must be reverted in an atomic manner.
TODO: Failed commitments and the implementation of rollbacks is tracked in #327 and #329.
Contents
- Home
- Persistence
- Changelog
-
Persistence
- Indexer
- Rpc
- Runtime
- State_Machine
-
Guides
- Roadmap
-
Guides
- Learning
- Guides
-
Guides
- Contributing
- Devlog
-
Guides
- Dependencies
-
Guides
- Releases
- Guides
- P2P
-
Shared
- Crypto
- Shared
-
Shared
- Modules
-
Build
- Config
- Consensus
-
Guides
- Telemetry
- Utility
- Logger