Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: expose BlockKeyCacheSize and enable WriteThrough datastore options #10614

Merged
merged 13 commits into from
Dec 19, 2024

Conversation

hsanjuan
Copy link
Contributor

@hsanjuan hsanjuan commented Dec 6, 2024

This enables WriteThrough blockstore and blockservice by default, exposing a new option and BlockKeyCacheSize to control two-queue cache and be able to disable it.

@hsanjuan hsanjuan self-assigned this Dec 6, 2024
@hsanjuan hsanjuan requested a review from a team as a code owner December 6, 2024 10:38
@hsanjuan
Copy link
Contributor Author

hsanjuan commented Dec 6, 2024

We can discuss a bit, because I am not sure when a non-writethrough blockstore/service makes fully sense. Probably on small datastores with very well primed caches that fit all keys and when re-writing the same keys all the time (not sure how common that is to warrant being the current default).

I don't think non-WriteThrough makes sense without bloom-filter. And bloom filter is primed on boot, which triggers reading all the keys (horrible in very large datastores).

  • flatfs: I'm guessing the OS provides some caching for folders etc. but Has() calls trigger 2-levels of directory listing on every block-write. Might be better than writing the block IF the block is already there.
  • Pebble/Badger: Given they can have 30x read amplification, I am not sure it ever makes sense to touch the read-path at all. They can optimize writes internally as they wish.

@hsanjuan hsanjuan force-pushed the fix/perf-pebble-improvements branch from 7f53a7d to cd96506 Compare December 6, 2024 10:51
core/coreapi/coreapi.go Outdated Show resolved Hide resolved
docs/changelogs/v0.33.md Outdated Show resolved Hide resolved
docs/config.md Outdated Show resolved Hide resolved
docs/config.md Outdated Show resolved Hide resolved
core/commands/dag/import.go Outdated Show resolved Hide resolved
@lidel lidel mentioned this pull request Dec 6, 2024
54 tasks
@hsanjuan hsanjuan force-pushed the fix/perf-pebble-improvements branch from 35cbba2 to b4b028c Compare December 10, 2024 11:23
@hsanjuan hsanjuan changed the title Feat: expose BlockKeyCacheSize and enable WriteThrough when bloom filter disabled Feat: expose BlockKeyCacheSize and enable WriteThrough datastore options Dec 10, 2024
@gammazero gammazero self-requested a review December 10, 2024 18:00
core/coreapi/coreapi.go Outdated Show resolved Hide resolved
config/datastore.go Outdated Show resolved Hide resolved
config/datastore.go Outdated Show resolved Hide resolved
@hsanjuan hsanjuan force-pushed the fix/perf-pebble-improvements branch from 0703a15 to 20c81ec Compare December 18, 2024 08:39
@hsanjuan hsanjuan force-pushed the fix/perf-pebble-improvements branch from 3eadcc3 to d3593a3 Compare December 18, 2024 08:58
@hsanjuan
Copy link
Contributor Author

Comments have been addressed, changelog has been improved, Boxo has been updated to main and nopfs too as flyby.

@hsanjuan hsanjuan requested a review from lidel December 18, 2024 08:59
@lidel lidel changed the title Feat: expose BlockKeyCacheSize and enable WriteThrough datastore options feat: expose BlockKeyCacheSize and enable WriteThrough datastore options Dec 19, 2024
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok to merge once #10631 lands in master + rebase + below refactor lands (I'll do that later today, just dropping comment so we don't forget)

core/coreapi/coreapi.go Outdated Show resolved Hide resolved
core/coreapi/unixfs.go Outdated Show resolved Hide resolved
core/node/core.go Outdated Show resolved Hide resolved
core/node/groups.go Outdated Show resolved Hide resolved
config/datastore.go Outdated Show resolved Hide resolved
config/init.go Outdated Show resolved Hide resolved
links need to be absolute because we reuse markdown in github releases
@lidel lidel enabled auto-merge (squash) December 19, 2024 22:45
@lidel lidel merged commit 519ae27 into master Dec 19, 2024
14 checks passed
@lidel lidel deleted the fix/perf-pebble-improvements branch December 19, 2024 23:12
@lidel lidel mentioned this pull request Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants