improve(benchmarks): support multiple values + visualizing runtimes #316

0xNineteen · 2024-10-17T10:36:01Z

docs in docs/benchmarks.md
scripts for visualizing scripts/view_bench.py
completely new benchmarking output from src/benchmarks.zig
add new tests to benchmarks
- fixes to leaks throughout the benchmarks (some require larger changes -- left for another PR)
changes to snapshot downloading (from previous PR/rex)
- closes account files after mmaping their data in - reducing open file limit errors (data is never modified so its ok) - this also lets us map it in with the PRIVATE flag
- reduce the accounts-db-snapshot benchmarks to only a single method which benchmarks loading and validation in one go
- improve snapshot download code

support multiple values from a benchmark

benchmark, read_time_min, read_time_max, read_time_mean, read_time_variance, benchmark, write_time_min, write_time_max, write_time_mean, write_time_variance, 
readWriteAccounts(100k accounts (1_slot - ram index - ram accounts)), 172156041, 158767959, 162868245, 15183799545214, 303852750, 286908417, 292925858, 39820330697776, 
readWriteAccounts(100k accounts (1_slot - disk index - ram accounts)), 165480250, 156170500, 160821658, 7611019088428, 319935833, 286708833, 304248199, 113169780175088,

NOTE: for multiple value outputs its not human readable at all - but i think thats ok -- most of our confirmations of speed ups/slow downs shouldnt be human checked, it should be computed and the increase/decrease should be human checked (which this PR allows for you to do easily)

support viewing all runtimes (`-r`)

{benchmark_name} ({field}), {runtime1}, {runtime2}, ...

readWriteAccounts(100k accounts (1_slot - ram index - ram accounts)) (read_time), 41451000, 40685750, 41123125, 40722417, 40743667
readWriteAccounts(100k accounts (1_slot - ram index - ram accounts)) (write_time), 81834042, 75340000, 76776125, 74969958, 74682792

visualizing all runtimes

./zig-out/bin/benchmark accounts_db_readwrite -r 2>&1 | tee b_results.txt # save output to file
python scripts/view_bench.py b_results.txt # view runtimes as a charts with one file source
python scripts/view_bench.py b_results.txt b_results2.txt # compare runtimes against two *equivalent* files

we support as many files as given from the cli

example output:

Viewing ['b_results.txt', 'b_results2.txt']
Saved to results/readWriteAccounts(100k accounts (1_slot - ram index - ram accounts)) (read_time).png
Saved to results/readWriteAccounts(100k accounts (1_slot - ram index - ram accounts)) (write_time).png
Saved to results/readWriteAccounts(100k accounts (1_slot - disk index - ram accounts)) (read_time).png
Saved to results/readWriteAccounts(100k accounts (1_slot - disk index - ram accounts)) (write_time).png
...

each point on y-axis=0 is a runtime
the point on y-axis=1 is the mean with the bar surrounding it being the standard deviation

note: builds off #289 -- i switched to mainly working on this feat so moved it to my own branch (instead of rexi's)

Sobeston

Seems good overall, but some issues:

Sobeston · 2024-10-23T18:01:50Z

docs/benchmarks.md

+
+```bash
+./zig-out/bin/benchmark accounts_db_readwrite -r 2>&1 | tee bench_results.txt # save output to file
+# NOTE: need to format doc to below
+python scripts/view_bench.py bench_results.txt # view runtimes as a charts with one file source
+python scripts/view_bench.py bench_results.txt b_results2.txt # compare runtimes against two *equivalent* files
+```
+


Just went to run this example and got a type error, leaving me without any data (?)

TypeError: unsupported operand type(s) for /: 'str' and 'int'

Sobeston · 2024-10-23T18:08:29Z

scripts/view_bench.py

+        for df_i, df in enumerate(dfs):
+            benchmark_runtimes = df.T[1:][i]
+            # convert to milliseconds 
+            if units == 'ms':
+                benchmark_runtimes = benchmark_runtimes / 1_000_000
+            if units == 's':
+                benchmark_runtimes = benchmark_runtimes / 1_000_000_000


this seems to be the source of the error in my previous comment

benchmark_runtimes = benchmark_runtimes / 1_000_000 ~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~

src/accountsdb/accounts_file.zig

Sobeston · 2024-10-23T18:15:13Z

src/accountsdb/db.zig

+        // defer logger.deinit();
+        // logger.spawn();


dangling comments

Sobeston · 2024-10-23T18:32:30Z

src/accountsdb/swiss_map.zig

-        });
-
-        return write_time;
+        // const read_faster_or_slower = if (read_speedup < 1.0) "slower" else "faster";


dangling comment

Sobeston · 2024-10-23T18:37:05Z

src/gossip/service.zig

+// test "benchmarkPullRequests" {
+//     _ = try BenchmarkGossipServicePullRequests.benchmarkPullRequests(.{
+//         .name = "1k_data_1k_pull_reqs",
+//         .n_data_populated = 10,
+//         .n_pull_requests = 2,
+//     });
+// }
+
+// test "benchmarkGossipService" {
+//     _ = try BenchmarkGossipServiceGeneral.benchmarkGossipService(.{
+//         .message_counts = .{
+//             .n_ping = 10,
+//             .n_push_message = 10,
+//             .n_pull_response = 10,
+//         },
+//     });
+// }


Are we fixing and re-enabling these, or removing these?

left a comment - right now they leak when run with the testing allocator -- will fix the leak in another PR since its a bit more involved to fix

c5d914b

https://github.com/orgs/Syndica/projects/2/views/10?pane=issue&itemId=84676970

Useful for when you need to build a specific executable, but not install it.

0xNineteen · 2024-10-24T16:00:37Z

discussed offline - changes to be included:

print human-readable format by default
output csv to a file (latest commit version & timestamp)

dnut assigned 0xNineteen Oct 17, 2024

0xNineteen marked this pull request as ready for review October 17, 2024 21:35

0xNineteen linked an issue Oct 17, 2024 that may be closed by this pull request

test(accountsdb): benchmark snapshot loading and validation #283

Open

0xNineteen force-pushed the 19/snapshot-bench branch from f91abb4 to 430d974 Compare October 18, 2024 16:09

0xNineteen requested review from dnut and Rexicon226 October 18, 2024 16:32

0xNineteen changed the title ~~feat(benchmarks): improve results~~ improve(benchmarks): support multiple values + visualizing runtimes Oct 22, 2024

0xNineteen requested a review from Sobeston October 23, 2024 09:42

Sobeston requested changes Oct 23, 2024

View reviewed changes

Rexicon226 and others added 21 commits October 24, 2024 09:32

build: add a no-run option

e4ac009

Useful for when you need to build a specific executable, but not install it.

speed up the snapshot downloader

12646ce

bench: add verifySnapshot benchmark

93e734d

download: return an error instead of panicing in writeCallback

a0932e9

free loading_threads if AccountsDB.init fails

d5b3800

benchmark progress

087a9c2

feat: csv benchmark outputs

cb6a32a

fix: lint and remove unused

add13ae

more fixes

44651b1

add variance to single output

a164340

fix: path allocations

1110fa7

re-enable read/write benchmarks

5ac8e8f

fix: formatting

8d65e74

fix: column headers

1a68cbf

fix: leaks in read/write test

8f0e7f8

fix benchmarks

18d3241

ci fix: dont run all the benchmarks

8df128a

attempt fix: OOM ci

046adea

CI check

98f9c0d

fix leak in CI

7b00106

fix and identify leaks

95b306f

0xNineteen added 14 commits October 24, 2024 09:33

fix lint

2ebae1a

feat: start agg print results

4bea9f6

feat: add option to output all runtimes

63292b1

remove extra bench

06772b4

add script to view benchmark runtimes

aff554a

more improvements on scripts

f1a8eb9

fix: update script for different lengths

d90cc87

fixes

b8a73ba

fix formatting

67ec58b

add docs

8fa6579

fix: lint

37f9726

reduce logger memory

f33e060

fix script

f0e9834

remove commented out code

c5d914b

0xNineteen force-pushed the 19/snapshot-bench branch from bb8257f to c5d914b Compare October 24, 2024 13:34

fix tests / build after merge

0965e37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve(benchmarks): support multiple values + visualizing runtimes #316

improve(benchmarks): support multiple values + visualizing runtimes #316

0xNineteen commented Oct 17, 2024 •

edited

Loading

Sobeston left a comment

Sobeston Oct 23, 2024

0xNineteen Oct 24, 2024 •

edited

Loading

Sobeston Oct 23, 2024

0xNineteen Oct 24, 2024 •

edited

Loading

Sobeston Oct 23, 2024

0xNineteen Oct 24, 2024 •

edited

Loading

Sobeston Oct 23, 2024

0xNineteen Oct 24, 2024 •

edited

Loading

Sobeston Oct 23, 2024

0xNineteen Oct 24, 2024 •

edited

Loading

0xNineteen commented Oct 24, 2024

improve(benchmarks): support multiple values + visualizing runtimes #316

Are you sure you want to change the base?

improve(benchmarks): support multiple values + visualizing runtimes #316

Conversation

0xNineteen commented Oct 17, 2024 • edited Loading

support multiple values from a benchmark

support viewing all runtimes (-r)

visualizing all runtimes

Sobeston left a comment

Choose a reason for hiding this comment

Sobeston Oct 23, 2024

Choose a reason for hiding this comment

0xNineteen Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Sobeston Oct 23, 2024

Choose a reason for hiding this comment

0xNineteen Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Sobeston Oct 23, 2024

Choose a reason for hiding this comment

0xNineteen Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Sobeston Oct 23, 2024

Choose a reason for hiding this comment

0xNineteen Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

Sobeston Oct 23, 2024

Choose a reason for hiding this comment

0xNineteen Oct 24, 2024 • edited Loading

Choose a reason for hiding this comment

0xNineteen commented Oct 24, 2024

0xNineteen commented Oct 17, 2024 •

edited

Loading

support viewing all runtimes (`-r`)

0xNineteen Oct 24, 2024 •

edited

Loading

0xNineteen Oct 24, 2024 •

edited

Loading

0xNineteen Oct 24, 2024 •

edited

Loading

0xNineteen Oct 24, 2024 •

edited

Loading

0xNineteen Oct 24, 2024 •

edited

Loading