Implement beacon db pruner #14687

dB2510 · 2024-12-02T19:11:19Z

What type of PR is this?
Feature

What does this PR do? Why is it needed?
This PR implements beacon db pruning based on weak subjectivity period. This is needed to avoid storing historical blocks beyond a point for non-archival nodes.

Which issues(s) does this PR fix?
#8787

Other notes for review
This PR implements a pruner service based on weak subjectivity period to periodically delete blocks and states beyond weak subjectivity period to maintain a constant-size database.

Acknowledgements

I have read CONTRIBUTING.md.
I have made an appropriate entry to CHANGELOG.md.
I have added a description to this PR with sufficient context for reviewers to understand this PR.

beacon-chain/db/pruner/pruner.go

prestonvanloon

Just one comment on the interface name.

The implementation looks good to me. Do you plan to plug it in with a feature flag in a follow up PR? Have you tested this?

beacon-chain/db/iface/interface.go

rkapka · 2024-12-13T17:50:47Z

No flag to customize the retention period?

dB2510 · 2024-12-16T08:26:31Z

@rkapka

No flag to customize the retention period?

Will include it in this PR.

beacon-chain/node/node.go

cmd/beacon-chain/flags/base.go

CHANGELOG.md

beacon-chain/db/kv/blocks.go

beacon-chain/db/pruner/pruner.go

beacon-chain/db/pruner/pruner_test.go

rkapka · 2024-12-30T17:46:27Z

Additional comment:

How about removing data for each bucket (or at least each major one) in parallel instead of block,state,block,state...? Using a wait group (I think) to make sure we wait for everything before the transaction completes. Before doing that we should check how long it takes to prune in the unhappy case. If it's a 100ms, then no need to optimize, but if it's 10s then it's worth it. The question then becomes what actually is the unhappy case? How many epochs of non-finality can the pruner expect? I am also wondering what happens when the pruner takes more than 12s to complete, as this will kick the next round of pruning before the previous has even completed. But that's probably a very unhappy case lol.

beacon-chain/db/kv/blocks.go

beacon-chain/db/kv/blocks_test.go

kasey

Overall you did a nice job on this, looking good!

Most of my comments are suggestions or even nitpicks, but I do think we should make the db query improvements - in particular using the slot from the index - before merging this.

CHANGELOG.md

beacon-chain/db/kv/blocks.go

beacon-chain/db/pruner/pruner.go

beacon-chain/sync/backfill/service.go

beacon-chain/db/pruner/pruner.go

Co-authored-by: Radosław Kapka <[email protected]>

Co-authored-by: Preston Van Loon <[email protected]>

Co-authored-by: Radosław Kapka <[email protected]>

kasey · 2025-01-22T23:03:40Z

beacon-chain/db/kv/blocks.go

 	bkt *bolt.Bucket,
 	startSlotEncoded, endSlotEncoded, startEpochEncoded, endEpochEncoded, slotStepEncoded interface{},
 ) ([][]byte, error) {
-	_, span := trace.StartSpan(ctx, "BeaconDB.blockRootsBySlotRange")


Why were these trace spans removed?

dB2510 force-pushed the beacondbpruning branch from 0138619 to df84c28 Compare December 5, 2024 15:40

rkapka reviewed Dec 10, 2024

View reviewed changes

beacon-chain/db/pruner/pruner.go Outdated Show resolved Hide resolved

prestonvanloon reviewed Dec 12, 2024

View reviewed changes

beacon-chain/db/iface/interface.go Outdated Show resolved Hide resolved

dB2510 force-pushed the beacondbpruning branch from 7eb6d40 to 20aafa7 Compare December 18, 2024 20:04

dB2510 changed the title ~~Implement weak subjectivity pruner~~ Implement beacon db pruner Dec 23, 2024

dB2510 force-pushed the beacondbpruning branch from 4f2164b to b373d0d Compare December 24, 2024 10:44

dB2510 marked this pull request as ready for review December 24, 2024 10:45

dB2510 requested a review from a team as a code owner December 24, 2024 10:45

dB2510 requested review from prestonvanloon, saolyn and terencechain December 24, 2024 10:45

terencechain reviewed Dec 24, 2024

View reviewed changes

beacon-chain/node/node.go Outdated Show resolved Hide resolved

prestonvanloon reviewed Dec 30, 2024

View reviewed changes

cmd/beacon-chain/flags/base.go Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

beacon-chain/db/kv/blocks.go Outdated Show resolved Hide resolved

rkapka reviewed Dec 30, 2024

View reviewed changes

rkapka reviewed Jan 7, 2025

View reviewed changes

beacon-chain/db/kv/blocks.go Outdated Show resolved Hide resolved

rkapka reviewed Jan 7, 2025

View reviewed changes

beacon-chain/db/kv/blocks.go Outdated Show resolved Hide resolved

rkapka reviewed Jan 7, 2025

View reviewed changes

beacon-chain/db/kv/blocks_test.go Show resolved Hide resolved

rkapka previously approved these changes Jan 9, 2025

View reviewed changes

kasey requested changes Jan 9, 2025

View reviewed changes

dB2510 dismissed rkapka’s stale review via b79ac4b January 15, 2025 11:54

dB2510 added 7 commits January 22, 2025 23:10

implement weak subjectivity pruner

2b11ca1

fix goimports

d51ae8e

add delete before slot method to database

ac8e5e1

add method to interface

e0fb890

update changelog

d4638c5

add flags

56c521c

wire pruner

c87d211

dB2510 and others added 18 commits January 22, 2025 23:11

align pruner with backfill service

1868a7b

rename db method

f501e8c

fix imports

5a78be2

delete block slot indices

187c95e

check for backfill and initial sync

77f7871

add tests

d983b0e

fix imports

d71d330

Update beacon-chain/db/kv/blocks.go

379aa2a

Co-authored-by: Radosław Kapka <[email protected]>

Update cmd/beacon-chain/flags/base.go

23b573f

Co-authored-by: Preston Van Loon <[email protected]>

Update beacon-chain/db/pruner/pruner.go

6e6f82a

Co-authored-by: Radosław Kapka <[email protected]>

cleanup

3c82247

fix buildkite

05272a5

initialise atomic bool

de2a209

delete data from remaining buckets

30dcd11

fix build

ee48748

fix build

424ad59

address review comments

7a64b6a

add test for blockParentRootIndicesBucket

d1a5c54

dB2510 force-pushed the beacondbpruning branch from 33b7bd4 to d1a5c54 Compare January 22, 2025 17:42

dB2510 added 2 commits January 22, 2025 23:26

fix changelog

1d9a4da

fix build

3b7b81d

kasey reviewed Jan 22, 2025

View reviewed changes

dB2510 added 3 commits January 23, 2025 13:50

address kasey's comments

768f68f

fix build

dbccc5f

add trace span to blockRootsBySlotRange

b9b84b9

kasey approved these changes Jan 23, 2025

View reviewed changes

dB2510 added this pull request to the merge queue Jan 28, 2025

Merged via the queue into develop with commit 4a9c60f Jan 28, 2025
15 checks passed

dB2510 deleted the beacondbpruning branch January 28, 2025 17:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement beacon db pruner #14687

Implement beacon db pruner #14687

dB2510 commented Dec 2, 2024 •

edited

Loading

prestonvanloon left a comment

rkapka commented Dec 13, 2024

dB2510 commented Dec 16, 2024 •

edited

Loading

rkapka commented Dec 30, 2024

kasey left a comment •

edited

Loading

kasey Jan 22, 2025

dB2510 Jan 23, 2025

Implement beacon db pruner #14687

Implement beacon db pruner #14687

Conversation

dB2510 commented Dec 2, 2024 • edited Loading

prestonvanloon left a comment

Choose a reason for hiding this comment

rkapka commented Dec 13, 2024

dB2510 commented Dec 16, 2024 • edited Loading

rkapka commented Dec 30, 2024

kasey left a comment • edited Loading

Choose a reason for hiding this comment

kasey Jan 22, 2025

Choose a reason for hiding this comment

dB2510 Jan 23, 2025

Choose a reason for hiding this comment

dB2510 commented Dec 2, 2024 •

edited

Loading

dB2510 commented Dec 16, 2024 •

edited

Loading

kasey left a comment •

edited

Loading