feat(blockstore): benchmarks #275

dadepo · 2024-09-18T06:24:52Z

Replicate mostly benches from

https://github.com/anza-xyz/agave/blob/master/ledger/benches/blockstore.rs
and
https://github.com/anza-xyz/agave/blob/master/ledger/benches/protobuf.rs

dadepo · 2024-09-30T11:24:48Z

src/ledger/tests.zig

+    pub const max_iterations = 5;
+
+    // Analogous to [bench_write_small](https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L59)
+    pub fn benchWriteSmall() !u64 {


Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchWriteSmall 5 757783 873809 2151500994 795654

Agave at 9c2098450ca7e5271e3690277992fbc910be27d0

running 1 test test bench_write_small ... bench: 23,708,904.20 ns/iter (+/- 222,018.81)

Are your benchmarks from release or debug builds of sig?

Here's what I'm seeing:

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchWriteSmall 5 49872 66281 29995977 57241

test bench_write_small ... bench: 17,406,472.70 ns/iter (+/- 8,623,383.67)

It was indeed a debug build.

Having a ReleaseSmall build I also get similar numbers as you got.

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchWriteSmall 5 30669 41452 13606829 34371

With iteration as 1000

zig build -Doptimize=ReleaseSafe benchmark -- ledger filtering benchmarks with prefix: ledger Benchmark Iterations Min(ns) Max(ns) Variance Mean(ns) --------------------------------------------------------------------------------- benchWriteSmall 1000 18309500 30636084 600023949574 18754731

dadepo · 2024-09-30T11:32:48Z

src/ledger/tests.zig

+    }
+
+    // Analogous to [bench_read_sequential]https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L78
+    pub fn benchReadSequential() !u64 {


NOTE: debug build

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadSequential 5 133011 156933 102519429 143209

Agave at 9c2098450ca7e5271e3690277992fbc910be27d0

running 1 test test bench_read_sequential ... bench: 2,740,116.70 ns/iter (+/- 304,996.16)

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadSequential 5 6110 8331 603665 7538

test bench_read_sequential ... bench: 2,403,107.92 ns/iter (+/- 584,953.52)

ReleaseSmall

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadSequential 5 3484 3816 12416 3680

With iteration as 1000

zig build -Doptimize=ReleaseSafe benchmark -- ledger filtering benchmarks with prefix: ledger Benchmark Iterations Min(ns) Max(ns) Variance Mean(ns) --------------------------------------------------------------------------------- benchReadSequential 1000 695250 3480542 23908758479 1006227

dadepo · 2024-09-30T11:37:34Z

src/ledger/tests.zig

+    }
+
+    // Analogous to [bench_read_random]https://github.com/anza-xyz/agave/blob/92eca1192b055d896558a78759d4e79ab4721ff1/ledger/benches/blockstore.rs#L103
+    pub fn benchReadRandom() !u64 {


Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadRandom 5 1993658 2326961 12944783078 2120888

Agave at 9c2098450ca7e5271e3690277992fbc910be27d0

running 1 test test bench_read_random ... bench: 2,820,841.70 ns/iter (+/- 28,719.65)

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadRandom 5 120573 132475 24537031 126483

test bench_read_random ... bench: 2,000,161.67 ns/iter (+/- 541,693.84)

The random performance is a lot worse than agave. The other benchmarks were much closer. It's strange, since the majority of what's being benchmarked here in both cases is rocksdb itself. I would have expected similar performance. Maybe the database needs to be tuned for random access.

I noticed that the agave benchmark reads 4369 shreds, whereas the sig benchmark reads 10499 indices. But that wouldn't explain a 50x slowdown. I also saw that in the agave benchmark, they only read 1/15 of the total number of shreds. Whereas in the sig benchmark, we read all of the shreds. But 4369 is not 1/15 of 10499, so something isn't adding up here.

It looks like benchReadSequential and benchReadRandom use the same input shreds in agave.blockstore.bench_read.shreds.bin. Is that intentional? Maybe each benchmark should use shreds that were generated by the respective agave benchmark, to ensure the data is comparable.

With releasemall, I also got

benchReadRandom 5 53996 72074 38589915 60359

it looks like benchReadSequential and benchReadRandom use the same input shreds in agave.blockstore.bench_read.shreds.bin. Is that intentional? Maybe each benchmark should use shreds that were generated by the respective agave benchmark, to ensure the data is comparable.

I'll look into this.

This is because the setup code for bother are identical. (As far as I can see)

https://github.com/anza-xyz/agave/blob/9c2098450ca7e5271e3690277992fbc910be27d0/ledger/benches/blockstore.rs#L79-L88

and

https://github.com/anza-xyz/agave/blob/9c2098450ca7e5271e3690277992fbc910be27d0/ledger/benches/blockstore.rs#L104-L113

Do you know what's causing this discrepancy?

I noticed that the agave benchmark reads 4369 shreds, whereas the sig benchmark reads 10499 indices. But that wouldn't explain a 50x slowdown. I also saw that in the agave benchmark, they only read 1/15 of the total number of shreds. Whereas in the sig benchmark, we read all of the shreds. But 4369 is not 1/15 of 10499, so something isn't adding up here.

I believe the discrepancy should be corrected with this commit 2a1dbc6

Running now, the result is:

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadRandom 5 1180 1483 11685 1276

With iteration as 1000

zig build -Doptimize=ReleaseSafe benchmark -- ledger filtering benchmarks with prefix: ledger Benchmark Iterations Min(ns) Max(ns) Variance Mean(ns) --------------------------------------------------------------------------------- benchReadRandom 1000 999917 4729250 59492207936 1295729

src/ledger/tests.zig

dnut · 2024-10-02T20:46:18Z

src/ledger/tests.zig

+    pub const max_iterations = 5;
+
+    // Analogous to [bench_write_small](https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L59)
+    pub fn benchWriteSmall() !u64 {


Are your benchmarks from release or debug builds of sig?

Here's what I'm seeing:

Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchWriteSmall 5 49872 66281 29995977 57241

test bench_write_small ... bench: 17,406,472.70 ns/iter (+/- 8,623,383.67)

dnut · 2024-10-02T20:46:37Z

src/ledger/tests.zig

+    }
+
+    // Analogous to [bench_read_sequential]https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L78
+    pub fn benchReadSequential() !u64 {


Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadSequential 5 6110 8331 603665 7538

test bench_read_sequential ... bench: 2,403,107.92 ns/iter (+/- 584,953.52)

dnut · 2024-10-02T20:47:39Z

src/ledger/tests.zig

+    }
+
+    // Analogous to [bench_read_random]https://github.com/anza-xyz/agave/blob/92eca1192b055d896558a78759d4e79ab4721ff1/ledger/benches/blockstore.rs#L103
+    pub fn benchReadRandom() !u64 {


Benchmark Iterations Min(us) Max(us) Variance Mean(us) --------------------------------------------------------------------------------- benchReadRandom 5 120573 132475 24537031 126483

test bench_read_random ... bench: 2,000,161.67 ns/iter (+/- 541,693.84)

The random performance is a lot worse than agave. The other benchmarks were much closer. It's strange, since the majority of what's being benchmarked here in both cases is rocksdb itself. I would have expected similar performance. Maybe the database needs to be tuned for random access.

I noticed that the agave benchmark reads 4369 shreds, whereas the sig benchmark reads 10499 indices. But that wouldn't explain a 50x slowdown. I also saw that in the agave benchmark, they only read 1/15 of the total number of shreds. Whereas in the sig benchmark, we read all of the shreds. But 4369 is not 1/15 of 10499, so something isn't adding up here.

It looks like benchReadSequential and benchReadRandom use the same input shreds in agave.blockstore.bench_read.shreds.bin. Is that intentional? Maybe each benchmark should use shreds that were generated by the respective agave benchmark, to ensure the data is comparable.

src/ledger/tests.zig

This PR removes the hardcoded gpa, with manual leak detection from the TestState used in tests. I believe the reason for this was to have more stack trace frames than what the std.testing.allocator offers, but this proved problematic when using the TestingState in other contexts like benchmarking. See #275 (comment)

dnut · 2024-10-16T21:36:57Z

src/ledger/benchmarks.zig

+    return rewards;
+}
+
+pub const BenchmarLegder = struct {


Suggested change

pub const BenchmarLegder = struct {

pub const BenchmarkLedger = struct {

dnut · 2024-10-18T13:25:20Z

I just want to highlight this comment in case you missed it: #275 (comment)

Also ledger is spelled with the "d" before the "g"

dadepo added 8 commits September 18, 2024 10:23

Added benchWriteSmall benchmark

2db1e1b

updated test data

b8ef631

Added benchReadSequential

e9eb195

Rename test file

baa2fd8

Added benchReadRandom

6f46e71

Set c allocator

fd8952a

typo

a7e6bfb

Check in test data

fe078fb

0xNineteen changed the title ~~feat(Blockstore): Blockstore benchmark~~ feat(blockstore): benchmarks Sep 23, 2024

dadepo added 5 commits September 29, 2024 14:56

Update test name

439100b

Reuse ledger.insert_shred.insertShredsForTest

7f3c380

Add benchSerializeWriteBincode

4d55ccd

Update title

8c80082

Add benchReadBincode

4528620

dadepo commented Sep 30, 2024

View reviewed changes

src/ledger/tests.zig Outdated Show resolved Hide resolved

dadepo marked this pull request as ready for review September 30, 2024 11:41

dnut self-requested a review October 1, 2024 19:42

dnut requested changes Oct 2, 2024

View reviewed changes

dadepo added 5 commits October 7, 2024 13:19

Moved ledger benchmark to own file

5fa8282

Typo fix

ab2086b

Fmt

077cda2

import via the root source

692d561

camelCase for functions

44fa660

dadepo mentioned this pull request Oct 7, 2024

refactor(ledger): remove manual leak detection in ledger tests #305

Merged

dadepo added 2 commits October 7, 2024 17:55

Merge branch 'main' into dade/blockstore-benchmark

cc8ac6c

Pass std.heap.c_allocator to TestState in ledger benchmark

dbcd012

dadepo added 2 commits October 7, 2024 18:38

Drop benchShreds in favour of re-using testShreds

b84cc5c

Remove unused imports

dcccd63

dadepo added 3 commits October 9, 2024 16:56

No need to put shreds into a tuple to later loop to deinit

aea8d1d

Merge branch 'main' into dade/blockstore-benchmark

7b04328

Switch to sig.time.Timer

9a1dfd9

dnut assigned dadepo Oct 11, 2024

dnut reviewed Oct 16, 2024

View reviewed changes

Fix typo

82a531d

dadepo added 10 commits October 19, 2024 22:29

Merge branch 'main' into dade/blockstore-benchmark

872a32b

typo

6249323

Fixes after merging.

6fdf8f3

Use num_reads sized random sample of indexes for benchReadRandom

2a1dbc6

Added comment

d684d62

Merge branch 'main' into dade/blockstore-benchmark

035963e

Fixes after merge

d8a9555

Switch to nanoseconds

5972a1e

Update iteration

f4eb228

Set iteration back to 5 for CI

1719072

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(blockstore): benchmarks #275

feat(blockstore): benchmarks #275

dadepo commented Sep 18, 2024 •

edited

Loading

dadepo Sep 30, 2024

dnut Oct 2, 2024

dadepo Oct 7, 2024 •

edited

Loading

dadepo Oct 24, 2024

dadepo Sep 30, 2024 •

edited

Loading

dnut Oct 2, 2024

dadepo Oct 7, 2024

dadepo Oct 24, 2024

dadepo Sep 30, 2024

dnut Oct 2, 2024

dadepo Oct 7, 2024

dadepo Oct 9, 2024

dnut Oct 16, 2024

dadepo Oct 19, 2024

dadepo Oct 24, 2024

dnut Oct 2, 2024

dnut Oct 2, 2024

dnut Oct 2, 2024

dnut Oct 16, 2024

dnut commented Oct 18, 2024

	pub const BenchmarLegder = struct {
	pub const BenchmarkLedger = struct {

feat(blockstore): benchmarks #275

Are you sure you want to change the base?

feat(blockstore): benchmarks #275

Conversation

dadepo commented Sep 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dadepo Oct 7, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dadepo Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dnut commented Oct 18, 2024

dadepo commented Sep 18, 2024 •

edited

Loading

dadepo Oct 7, 2024 •

edited

Loading

dadepo Sep 30, 2024 •

edited

Loading