change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process #8596

arya2 · 2024-06-07T04:28:16Z

Motivation

We want to use the ChainTipChange struct and access blocks in Zebra's non-finalized best chain from zebra-scan while it runs as a separate process.

Closes #8580.
Closes #8576.

PR Author Checklist

Check before marking the PR as ready for review:

Will the PR name make sense to users?
Does the PR have a priority label?
Have you added or updated tests?
Is the documentation up to date?

For significant changes:

Is there a summary in the CHANGELOG?
Can these changes be split into multiple PRs?

If a checkbox isn't relevant to the PR, mark it as done.

Solution

Adds a TrustedChainSync struct for:

Syncing non-finalized blocks from a Zebra RPC server,
Updating the ChainTipSender when a new block has been committed to the non-finalized state, and
Updating the ChainTipSender when the finalized tip changes

Testing

A new test was added to check that the syncer behaves as expected when:

the best chain tip is in the finalized state
the best chain tip is in the non-finalized state
there's a chain reorganization and a side chain becomes the new best chain

Review

Anyone can review.

Reviewer Checklist

Check before approving the PR:

Does the PR scope match the ticket?
Are there enough tests to make sure it works? Do the tests cover the PR motivation?
Are all the PR blockers dealt with?
PR blockers can be dealt with in new tickets or PRs.

And check the PR Author checklist is complete.

…bool to try enabling elasticsearch

…hain tip sender when the finalized tip has changed.

…yre type

…g a block unless it has the missing block error code.

…ain grows

…uests

…d-only db

oxarbitrage

looks really good, thank you.

zebra-rpc/src/sync.rs

mpguerra · 2024-07-04T09:47:23Z

@Mergifyio refresh

mergify · 2024-07-04T09:47:26Z

refresh

✅ Pull request refreshed

mpguerra · 2024-07-04T10:20:48Z

@arya2 seems like zebra is failing to build https://github.com/ZcashFoundation/zebra/actions/runs/9767174563/job/27038729170?pr=8596
Can you please take a look? It would be good to get this merged and close off the related issues

…crate

oxarbitrage · 2024-07-08T20:37:57Z

Re-running by a CI failure in regtest_submit_blocks windows version test: https://github.com/ZcashFoundation/zebra/actions/runs/9845868582/job/27182357737?pr=8596

arya2 added 10 commits June 6, 2024 20:18

Adds an init_read_only() fn in zebra-state

26c0e5d

moves elasticsearch initialization to FinalizedState::new_with_debug()

281668b

Updates callers of FinalizedState::{new, new_with_debug} to pass a …

33a3ccf

…bool to try enabling elasticsearch

Adds a non-finalized read state syncer to zebra-rpc

995811e

moves, removes, updates, or addresses TODOs

f94bd41

reduces disk IO while waiting for the a new chain tip & updates the c…

d57c50a

…hain tip sender when the finalized tip has changed.

Returns boxed errors from RpcRequestClient methods instead of color_e…

8d246c3

…yre type

Avoids resetting the non-finalized state when there's an error gettin…

5470979

…g a block unless it has the missing block error code.

Adds stub for acceptance test(s) and removes outdated TODO

0d35fc3

adds TODOs for testing

0d1f434

arya2 added A-rpc Area: Remote Procedure Call interfaces A-state Area: State / database changes P-Medium ⚡ labels Jun 7, 2024

arya2 self-assigned this Jun 7, 2024

This was linked to issues Jun 7, 2024

Add a function for syncing blocks from Zebra's non-finalized best chain to a ReadStateService in another process #8576

Closed

Add a current_tip() or similar to ReadState #8580

Closed

arya2 added 3 commits June 10, 2024 21:22

Tests that ChainTipChange is updated when the non-finalized best ch…

dc514b8

…ain grows

adds a last_chain_tip_hash and uses a FuturesOrdered for getblock req…

302facf

…uests

Fixes a pre-flush sync issue by using a secondary db instead of a rea…

a1ad80f

…d-only db

Base automatically changed from init-read-state to main June 12, 2024 13:35

arya2 added 6 commits June 13, 2024 18:09

Moves disk IO to blocking tasks

c9319e0

Updates acceptance test to how forks are handled

9d2d1df

Checks synced read state for all of the expected blocks

d31526e

checks that there isn't a tip change until the best chain changes

f18344f

checks for chain tip changes in test

112294f

Merge branch 'main' into non-final-read-state

ac62bae

arya2 marked this pull request as ready for review June 14, 2024 03:06

arya2 requested review from a team as code owners June 14, 2024 03:06

arya2 removed the request for review from a team June 14, 2024 03:06

arya2 requested a review from oxarbitrage June 19, 2024 23:31

Merge branch 'main' into non-final-read-state

e642876

oxarbitrage previously approved these changes Jun 20, 2024

View reviewed changes

zebra-rpc/src/sync.rs Outdated Show resolved Hide resolved

zebra-rpc/src/sync.rs Outdated Show resolved Hide resolved

oxarbitrage mentioned this pull request Jun 20, 2024

Use sync::init_read_state_with_syncer() in the zebra-scanner #8633

Closed

2 tasks

Updates POLL_DELAY documentation

8fb798b

arya2 dismissed oxarbitrage’s stale review via 8fb798b June 20, 2024 14:15

arya2 added 2 commits June 20, 2024 10:16

Updates method docs

f0b5ee5

Fixes a test bug

2743051

arya2 mentioned this pull request Jun 20, 2024

feat(scanner): Add a new zebra-scanner binary #8608

Merged

7 tasks

arya2 requested a review from oxarbitrage June 20, 2024 22:15

Merge branch 'main' into non-final-read-state

0b875ba

oxarbitrage mentioned this pull request Jun 28, 2024

tests(scanner): Move zebra scanner tests to binary #8659

Merged

7 tasks

arya2 removed the do-not-merge Tells Mergify not to merge this PR label Jul 2, 2024

Merge branch 'main' into non-final-read-state

abe0af7

conradoplg previously approved these changes Jul 2, 2024

View reviewed changes

use rpc-client feature in zebrad production code

8391ae7

oxarbitrage dismissed conradoplg’s stale review via 8391ae7 July 8, 2024 18:08

use rpc-client feature in zebra-node-services for building zebra-rpc …

d996cc6

…crate

mpguerra previously approved these changes Jul 8, 2024

View reviewed changes

Merge branch 'main' into non-final-read-state

92db104

arya2 mentioned this pull request Jul 8, 2024

fix(tests): Address a port conflict issue in the regtest_submit_blocks test #8665

Merged

7 tasks

Merge remote-tracking branch 'origin/main' into non-final-read-state

0e25311

arya2 dismissed mpguerra’s stale review via 0e25311 July 8, 2024 23:51

oxarbitrage approved these changes Jul 9, 2024

View reviewed changes

mergify bot merged commit 4213e82 into main Jul 9, 2024
132 checks passed

mergify bot deleted the non-final-read-state branch July 9, 2024 14:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process #8596

change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process #8596

arya2 commented Jun 7, 2024 •

edited

Loading

oxarbitrage left a comment

mpguerra commented Jul 4, 2024

mergify bot commented Jul 4, 2024

mpguerra commented Jul 4, 2024

oxarbitrage commented Jul 8, 2024

change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process #8596

change(rpc): Adds a TrustedChainSync struct for keeping up with Zebra's non-finalized best chain from a separate process #8596

Conversation

arya2 commented Jun 7, 2024 • edited Loading

Motivation

PR Author Checklist

Check before marking the PR as ready for review:

For significant changes:

Solution

Testing

Review

Reviewer Checklist

oxarbitrage left a comment

Choose a reason for hiding this comment

mpguerra commented Jul 4, 2024

mergify bot commented Jul 4, 2024

✅ Pull request refreshed

mpguerra commented Jul 4, 2024

oxarbitrage commented Jul 8, 2024

arya2 commented Jun 7, 2024 •

edited

Loading