-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(blockstore): benchmarks #275
base: main
Are you sure you want to change the base?
Conversation
src/ledger/tests.zig
Outdated
pub const max_iterations = 5; | ||
|
||
// Analogous to [bench_write_small](https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L59) | ||
pub fn benchWriteSmall() !u64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchWriteSmall 5 757783 873809 2151500994 795654
Agave at 9c2098450ca7e5271e3690277992fbc910be27d0
running 1 test
test bench_write_small ... bench: 23,708,904.20 ns/iter (+/- 222,018.81)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are your benchmarks from release or debug builds of sig?
Here's what I'm seeing:
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchWriteSmall 5 49872 66281 29995977 57241
test bench_write_small ... bench: 17,406,472.70 ns/iter (+/- 8,623,383.67)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It was indeed a debug build.
Having a ReleaseSmall
build I also get similar numbers as you got.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchWriteSmall 5 30669 41452 13606829 34371
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With iteration as 1000
zig build -Doptimize=ReleaseSafe benchmark -- ledger
filtering benchmarks with prefix: ledger
Benchmark Iterations Min(ns) Max(ns) Variance Mean(ns)
---------------------------------------------------------------------------------
benchWriteSmall 1000 18309500 30636084 600023949574 18754731
src/ledger/tests.zig
Outdated
} | ||
|
||
// Analogous to [bench_read_sequential]https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L78 | ||
pub fn benchReadSequential() !u64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NOTE: debug build
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadSequential 5 133011 156933 102519429 143209
Agave at 9c2098450ca7e5271e3690277992fbc910be27d0
running 1 test
test bench_read_sequential ... bench: 2,740,116.70 ns/iter (+/- 304,996.16)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadSequential 5 6110 8331 603665 7538
test bench_read_sequential ... bench: 2,403,107.92 ns/iter (+/- 584,953.52)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ReleaseSmall
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadSequential 5 3484 3816 12416 3680
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With iteration as 1000
zig build -Doptimize=ReleaseSafe benchmark -- ledger
filtering benchmarks with prefix: ledger
Benchmark Iterations Min(ns) Max(ns) Variance Mean(ns)
---------------------------------------------------------------------------------
benchReadSequential 1000 695250 3480542 23908758479 1006227
src/ledger/tests.zig
Outdated
} | ||
|
||
// Analogous to [bench_read_random]https://github.com/anza-xyz/agave/blob/92eca1192b055d896558a78759d4e79ab4721ff1/ledger/benches/blockstore.rs#L103 | ||
pub fn benchReadRandom() !u64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadRandom 5 1993658 2326961 12944783078 2120888
Agave at 9c2098450ca7e5271e3690277992fbc910be27d0
running 1 test
test bench_read_random ... bench: 2,820,841.70 ns/iter (+/- 28,719.65)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadRandom 5 120573 132475 24537031 126483
test bench_read_random ... bench: 2,000,161.67 ns/iter (+/- 541,693.84)
The random performance is a lot worse than agave. The other benchmarks were much closer. It's strange, since the majority of what's being benchmarked here in both cases is rocksdb itself. I would have expected similar performance. Maybe the database needs to be tuned for random access.
I noticed that the agave benchmark reads 4369 shreds, whereas the sig benchmark reads 10499 indices. But that wouldn't explain a 50x slowdown. I also saw that in the agave benchmark, they only read 1/15 of the total number of shreds. Whereas in the sig benchmark, we read all of the shreds. But 4369 is not 1/15 of 10499, so something isn't adding up here.
It looks like benchReadSequential and benchReadRandom use the same input shreds in agave.blockstore.bench_read.shreds.bin
. Is that intentional? Maybe each benchmark should use shreds that were generated by the respective agave benchmark, to ensure the data is comparable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With releasemall, I also got
benchReadRandom 5 53996 72074 38589915 60359
it looks like benchReadSequential and benchReadRandom use the same input shreds in agave.blockstore.bench_read.shreds.bin. Is that intentional? Maybe each benchmark should use shreds that were generated by the respective agave benchmark, to ensure the data is comparable.
I'll look into this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is because the setup code for bother are identical. (As far as I can see)
and
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you know what's causing this discrepancy?
I noticed that the agave benchmark reads 4369 shreds, whereas the sig benchmark reads 10499 indices. But that wouldn't explain a 50x slowdown. I also saw that in the agave benchmark, they only read 1/15 of the total number of shreds. Whereas in the sig benchmark, we read all of the shreds. But 4369 is not 1/15 of 10499, so something isn't adding up here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe the discrepancy should be corrected with this commit 2a1dbc6
Running now, the result is:
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadRandom 5 1180 1483 11685 1276
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With iteration as 1000
zig build -Doptimize=ReleaseSafe benchmark -- ledger
filtering benchmarks with prefix: ledger
Benchmark Iterations Min(ns) Max(ns) Variance Mean(ns)
---------------------------------------------------------------------------------
benchReadRandom 1000 999917 4729250 59492207936 1295729
src/ledger/tests.zig
Outdated
pub const max_iterations = 5; | ||
|
||
// Analogous to [bench_write_small](https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L59) | ||
pub fn benchWriteSmall() !u64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are your benchmarks from release or debug builds of sig?
Here's what I'm seeing:
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchWriteSmall 5 49872 66281 29995977 57241
test bench_write_small ... bench: 17,406,472.70 ns/iter (+/- 8,623,383.67)
src/ledger/tests.zig
Outdated
} | ||
|
||
// Analogous to [bench_read_sequential]https://github.com/anza-xyz/agave/blob/cfd393654f84c36a3c49f15dbe25e16a0269008d/ledger/benches/blockstore.rs#L78 | ||
pub fn benchReadSequential() !u64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadSequential 5 6110 8331 603665 7538
test bench_read_sequential ... bench: 2,403,107.92 ns/iter (+/- 584,953.52)
src/ledger/tests.zig
Outdated
} | ||
|
||
// Analogous to [bench_read_random]https://github.com/anza-xyz/agave/blob/92eca1192b055d896558a78759d4e79ab4721ff1/ledger/benches/blockstore.rs#L103 | ||
pub fn benchReadRandom() !u64 { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark Iterations Min(us) Max(us) Variance Mean(us)
---------------------------------------------------------------------------------
benchReadRandom 5 120573 132475 24537031 126483
test bench_read_random ... bench: 2,000,161.67 ns/iter (+/- 541,693.84)
The random performance is a lot worse than agave. The other benchmarks were much closer. It's strange, since the majority of what's being benchmarked here in both cases is rocksdb itself. I would have expected similar performance. Maybe the database needs to be tuned for random access.
I noticed that the agave benchmark reads 4369 shreds, whereas the sig benchmark reads 10499 indices. But that wouldn't explain a 50x slowdown. I also saw that in the agave benchmark, they only read 1/15 of the total number of shreds. Whereas in the sig benchmark, we read all of the shreds. But 4369 is not 1/15 of 10499, so something isn't adding up here.
It looks like benchReadSequential and benchReadRandom use the same input shreds in agave.blockstore.bench_read.shreds.bin
. Is that intentional? Maybe each benchmark should use shreds that were generated by the respective agave benchmark, to ensure the data is comparable.
This PR removes the hardcoded gpa, with manual leak detection from the TestState used in tests. I believe the reason for this was to have more stack trace frames than what the std.testing.allocator offers, but this proved problematic when using the TestingState in other contexts like benchmarking. See #275 (comment)
This PR removes the hardcoded gpa, with manual leak detection from the TestState used in tests. I believe the reason for this was to have more stack trace frames than what the std.testing.allocator offers, but this proved problematic when using the TestingState in other contexts like benchmarking. See #275 (comment)
src/ledger/benchmarks.zig
Outdated
return rewards; | ||
} | ||
|
||
pub const BenchmarLegder = struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pub const BenchmarLegder = struct { | |
pub const BenchmarkLedger = struct { |
I just want to highlight this comment in case you missed it: #275 (comment) Also ledger is spelled with the "d" before the "g" |
Replicate mostly benches from
https://github.com/anza-xyz/agave/blob/master/ledger/benches/blockstore.rs
and
https://github.com/anza-xyz/agave/blob/master/ledger/benches/protobuf.rs