fix(p2p): cache responses to serve without roundtrip to db #2352

rymnc · 2024-10-14T15:45:42Z

Linked Issues/PRs

the intermittent outages on testnet

Description

When we request transactions for a given block range, we shouldn't only keep using the same peer causing pressure on it. we should pick a random one with the same height and try to get the transactions from that instead.

This PR caches p2p responses (ttl 10 seconds by default) and serves requests from cache falling back to db for others.

Checklist

Breaking changes are clearly marked as such in the PR description and changelog
New behavior is reflected in tests
The specification matches the implemented behavior (link update PR if changes are needed)

Before requesting review

I have reviewed the code myself
I have created follow-up issues caused by this PR and linked them here

After merging, notify other teams

[Add or remove entries as needed]

Rust SDK
Sway compiler
Platform documentation (for out-of-organization contributors, the person merging the PR will do this)
Someone else?

rymnc · 2024-10-15T14:53:42Z

crates/services/p2p/src/service.rs

+impl CachedView {
+    fn new(metrics: bool) -> Self {
+        Self {
+            sealed_block_headers: DashMap::new(),
+            transactions_on_blocks: DashMap::new(),
+            metrics,
+        }
+    }


we probably want to also support sub-ranges or even partial ranges here, but can be in the future :)

netrome

Thanks for implementing this. I Hate to be annoying here, but to approve this I need to:

See the Changelog updated.
Understand the reasoning behind the current caching strategy and the benefits/drawbacks over an LRU cache.
Be certain that we don't open the door to OOM attacks by allowing our cache to be overloaded.

Let me know your thoughts on 2 and 3. I'm happy to jump on a call to discuss this and figure out a good path forward.

CHANGELOG.md

crates/services/p2p/src/service.rs

netrome · 2024-10-15T19:01:34Z

crates/services/p2p/src/cached_view.rs

+pub struct CachedView {
+    sealed_block_headers: DashMap<Range<u32>, Vec<SealedBlockHeader>>,
+    transactions_on_blocks: DashMap<Range<u32>, Vec<Transactions>>,
+    metrics: bool,
+}


I'm a bit hesitant to the current approach of storing everything and clearing on a regular interval. Right now, there is no memory limit of the cache, and we use ranges as keys. So if someone queries the ranges (1..=4, 1..=2, 3..=4), we'd store all blocks in the 1..=4 range twice - and this could theoretically grow quadratically for larger ranges.

I would assume that the most popular queries at a given time are quite similar. Why not use a normal LRU cache with fixed memory size? Alternatively just maintain a cache over the last $N$ block headers and their transactions, evicting old ones as new ones gets populated?

yup, its still wip.

Ah right, I see this PR is still a draft :)

we now use block height as the key in 6422210

we will retain the time-based eviction strategy for now

… instead of a per-range basis

rymnc · 2024-10-16T06:35:34Z

synced a testnet node and had 2 local nodes sync from it at the same time -

Co-authored-by: Mårten Blankfors <[email protected]>

rafal-ch

~~Partial review~~

Review completed.

crates/services/p2p/src/cached_view.rs

rafal-ch · 2024-10-30T11:49:49Z

crates/services/p2p/src/cached_view.rs

+        let mut items = Vec::new();
+        let mut missing_start = None;
+
+        for height in range.clone() {


It came as a surprise that Range isn't Copy, but seems like it's not: rust-lang/rust#21846 (comment)

I think you can go around it by copying the range bounds before consuming the range itself, since those are Copy.

e.g.

let range_copy = range.start..range.end for height in range { /* ... */ } // No need to clone anymore

In this case you only need range.end, so it might be better to copy that value only

crates/services/p2p/src/service.rs

rafal-ch

LGTM 👍

Co-authored-by: Rafał Chabowski <[email protected]>

xgreenx · 2024-11-13T21:26:29Z

crates/services/p2p/src/cached_view.rs

+        }
+    }
+
+    pub(super) fn clear(&self) {


Maybe we could use some LRU cache instead of wiping the whole cache every 10 seconds? This is just a thought since the current approach should also work.

Yeah, in the future we can use an LRU :) we discussed it somewhere above too

xgreenx · 2024-11-13T21:27:26Z

crates/services/p2p/src/service.rs

+    cache_reset_interval: Duration,
+    next_cache_reset_time: Instant,


Looks like it could be an internal logic of the CachedView and on each insert/get we can clean it.

it could be, but I wanted to not make the get_from_cache_or_db require a mutable reference to Self because it's just a getter. no strong opinion here, so if you want it that way, i can move it around

xgreenx · 2024-11-13T21:43:06Z

crates/services/p2p/src/cached_view.rs

+
+        for height in range.clone() {
+            if let Some(item) = cache.get(&height) {
+                items.push(item.clone());


It can be follow up PR, but it would be nice if we avoid heavy clone here and used Arc instead

added comment here - d897cba

associated issue: #2436

acerone85 · 2024-11-14T10:17:53Z

crates/services/p2p/src/cached_view.rs

+        let block_height_range = 0..100;
+        let sealed_headers = default_sealed_headers(block_height_range.clone());
+        let result = cached_view
+            .get_sealed_headers(&db, block_height_range.clone())


I would expect the cache to be linked to the DB at the time it is created, rather than having to specify the DB when invoking the function get_sealed_headers or get_transactions. Just curious to know what's the reason behind this choice?

you will notice that the view of the current tip of the db (LatestView) is passed into the CachedView while making calls

acerone85 · 2024-11-14T10:32:17Z

LGTM. I have a side question of whether the cache should be cleared in case of a DB rollback, to avoid inconsistencies?

rymnc · 2024-11-14T10:34:24Z

LGTM. I have a side question of whether the cache should be cleared in case of a DB rollback, to avoid inconsistencies?

that's a good question! i wonder if we have a hook from the db to be notified when it gets rolled back.

rymnc requested review from xgreenx, Dentosal and MitchTurner as code owners October 14, 2024 15:45

rymnc requested a review from a team October 14, 2024 15:47

rymnc changed the title ~~fix(p2p): get transactions from a random peer with the same height~~ fix(p2p): cache responses to serve without roundtrip to db Oct 14, 2024

rymnc added 3 commits October 15, 2024 01:52

chore: changelog

ba6bdab

test: arc mutex on cachedview, reset every 10 sec

5a51e0c

fix: err

8079739

rymnc force-pushed the fix/p2p-round-robin branch from dcdaaf7 to 8079739 Compare October 14, 2024 20:22

rymnc marked this pull request as draft October 14, 2024 21:37

rymnc added 2 commits October 15, 2024 03:25

fix(p2p): use dashmap, remove mutex, remove instantiation with view

19a32f3

fix: fmt

805a866

rymnc self-assigned this Oct 14, 2024

rymnc added the fuel-p2p label Oct 14, 2024

rymnc added 4 commits October 15, 2024 05:42

Merge branch 'master' into fix/p2p-round-robin

fd0aea0

chore: add metrics to cache hits/misses

c5db7a6

fix: clippy

b4763be

Merge branch 'master' into fix/p2p-round-robin

b6bbf61

rymnc commented Oct 15, 2024

View reviewed changes

chore: refactor cached_view into own module

5b03fa0

rymnc force-pushed the fix/p2p-round-robin branch from 009dc14 to 5b03fa0 Compare October 15, 2024 15:33

netrome requested changes Oct 15, 2024

View reviewed changes

chore: retain time based clear, but cache is now on a per block basis…

6422210

… instead of a per-range basis

fix: metrics logging and clearing

2b2a8fb

netrome mentioned this pull request Oct 16, 2024

Keep data in fails cases in sync service #2361

Open

5 tasks

rymnc and others added 3 commits October 28, 2024 16:00

Merge branch 'master' into fix/p2p-round-robin

0a3724d

fix: fmt and clippy

f597ac9

Update CHANGELOG.md

25515ee

Co-authored-by: Mårten Blankfors <[email protected]>

rymnc requested a review from netrome October 29, 2024 09:39

rymnc dismissed netrome’s stale review via f16cb2f October 30, 2024 08:43

netrome previously approved these changes Oct 30, 2024

View reviewed changes

rafal-ch reviewed Oct 30, 2024

View reviewed changes

Merge branch 'master' into fix/p2p-round-robin

c6f00af

rafal-ch reviewed Oct 30, 2024

View reviewed changes

crates/services/p2p/src/service.rs Outdated Show resolved Hide resolved

crates/services/p2p/src/service.rs Outdated Show resolved Hide resolved

fix: add test

1630e2b

rymnc dismissed netrome’s stale review via 1630e2b October 30, 2024 13:04

rymnc requested review from rafal-ch and netrome October 30, 2024 13:04

netrome previously approved these changes Oct 30, 2024

View reviewed changes

rafal-ch previously approved these changes Oct 30, 2024

View reviewed changes

rymnc added 2 commits October 30, 2024 19:41

Merge branch 'master' into fix/p2p-round-robin

22d09cf

fix: call run multiple times until heartbeat report is available

09d7bd2

rymnc dismissed stale reviews from rafal-ch and netrome via 09d7bd2 October 30, 2024 15:51

rymnc added 3 commits October 30, 2024 17:07

fix: clippy

3972a47

fix: dont allow cache reset interval to swallow runtime of other tasks

69de071

Merge branch 'master' into fix/p2p-round-robin

c634764

rymnc requested review from netrome and rafal-ch October 31, 2024 04:16

rymnc mentioned this pull request Oct 31, 2024

P2P is doing a lot of database lookups #2023

Open

rymnc linked an issue Oct 31, 2024 that may be closed by this pull request

P2P is doing a lot of database lookups #2023

Open

chore: visibility of CachedView

7a0a776

Co-authored-by: Rafał Chabowski <[email protected]>

xgreenx reviewed Nov 13, 2024

View reviewed changes

acerone85 reviewed Nov 14, 2024

View reviewed changes

rymnc mentioned this pull request Nov 14, 2024

chore: replace expensive clones in p2p caching with Arc #2436

Open

1 task

chore: todos

d897cba

rymnc requested a review from acerone85 November 14, 2024 10:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(p2p): cache responses to serve without roundtrip to db #2352

fix(p2p): cache responses to serve without roundtrip to db #2352

rymnc commented Oct 14, 2024 •

edited

Loading

rymnc Oct 15, 2024

netrome left a comment

netrome Oct 15, 2024

rymnc Oct 15, 2024

netrome Oct 15, 2024

rymnc Oct 16, 2024

rymnc commented Oct 16, 2024

rafal-ch left a comment •

edited

Loading

rafal-ch Oct 30, 2024

acerone85 Nov 14, 2024

rafal-ch left a comment

xgreenx Nov 13, 2024

rymnc Nov 14, 2024

xgreenx Nov 13, 2024

rymnc Nov 14, 2024

xgreenx Nov 13, 2024

rymnc Nov 14, 2024

rymnc Nov 14, 2024

acerone85 Nov 14, 2024

rymnc Nov 14, 2024

acerone85 commented Nov 14, 2024

rymnc commented Nov 14, 2024

		cache_reset_interval: Duration,
		next_cache_reset_time: Instant,

fix(p2p): cache responses to serve without roundtrip to db #2352

Are you sure you want to change the base?

fix(p2p): cache responses to serve without roundtrip to db #2352

Conversation

rymnc commented Oct 14, 2024 • edited Loading

Linked Issues/PRs

Description

Checklist

Before requesting review

After merging, notify other teams

Choose a reason for hiding this comment

netrome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rymnc commented Oct 16, 2024

rafal-ch left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rafal-ch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acerone85 commented Nov 14, 2024

rymnc commented Nov 14, 2024

rymnc commented Oct 14, 2024 •

edited

Loading

rafal-ch left a comment •

edited

Loading