Revamp the benchmark suite #606
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There is a large number of outstanding performance PRs that I want to
merge, but we need a decent benchmark to judge if they are effective.
I went to borrow rapidjson's benchmark suite, which is a good start.
I only kept the comparison with Oj and RapidJSON, because YAJL is
slower on most benchmarks, so little point comparing to it.
Encoding:
```
== Encoding small nested array (121 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 88.225k i/100ms
oj 209.862k i/100ms
rapidjson 128.978k i/100ms
Calculating -------------------------------------
json 914.611k (± 0.4%) i/s (1.09 μs/i) - 4.588M in 5.016099s
oj 2.163M (± 0.2%) i/s (462.39 ns/i) - 10.913M in 5.045964s
rapidjson 1.392M (± 1.3%) i/s (718.55 ns/i) - 6.965M in 5.005438s
Comparison:
json: 914610.6 i/s
oj: 2162693.5 i/s - 2.36x faster
rapidjson: 1391682.6 i/s - 1.52x faster
== Encoding small hash (65 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 142.093k i/100ms
oj 651.412k i/100ms
rapidjson 237.706k i/100ms
Calculating -------------------------------------
json 1.478M (± 0.7%) i/s (676.78 ns/i) - 7.389M in 5.000866s
oj 7.150M (± 0.7%) i/s (139.85 ns/i) - 35.828M in 5.010756s
rapidjson 2.250M (± 1.6%) i/s (444.46 ns/i) - 11.410M in 5.072451s
Comparison:
json: 1477595.1 i/s
oj: 7150472.0 i/s - 4.84x faster
rapidjson: 2249926.7 i/s - 1.52x faster
== Encoding twitter.json (466906 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 101.000 i/100ms
oj 223.000 i/100ms
rapidjson 105.000 i/100ms
Calculating -------------------------------------
json 1.017k (± 0.7%) i/s (982.83 μs/i) - 5.151k in 5.062786s
oj 2.244k (± 0.7%) i/s (445.72 μs/i) - 11.373k in 5.069428s
rapidjson 1.069k (± 4.6%) i/s (935.20 μs/i) - 5.355k in 5.016652s
Comparison:
json: 1017.5 i/s
oj: 2243.6 i/s - 2.21x faster
rapidjson: 1069.3 i/s - same-ish: difference falls within error
== Encoding citm_catalog.json (500299 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 77.000 i/100ms
oj 129.000 i/100ms
rapidjson 96.000 i/100ms
Calculating -------------------------------------
json 767.217 (± 2.5%) i/s (1.30 ms/i) - 3.850k in 5.021957s
oj 1.291k (± 1.5%) i/s (774.45 μs/i) - 6.579k in 5.096439s
rapidjson 959.527 (± 1.1%) i/s (1.04 ms/i) - 4.800k in 5.003052s
Comparison:
json: 767.2 i/s
oj: 1291.2 i/s - 1.68x faster
rapidjson: 959.5 i/s - 1.25x faster
== Encoding canada.json (2090234 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 1.000 i/100ms
oj 3.000 i/100ms
rapidjson 1.000 i/100ms
Calculating -------------------------------------
json 19.748 (± 0.0%) i/s (50.64 ms/i) - 99.000 in 5.013336s
oj 31.016 (± 0.0%) i/s (32.24 ms/i) - 156.000 in 5.029732s
rapidjson 19.419 (± 0.0%) i/s (51.50 ms/i) - 98.000 in 5.050382s
Comparison:
json: 19.7 i/s
oj: 31.0 i/s - 1.57x faster
rapidjson: 19.4 i/s - 1.02x slower
== Encoding many #to_json calls (2661 bytes)
oj does not match expected output. Skipping
rapidjson unsupported (Invalid object key type: Object)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 2.129k i/100ms
Calculating -------------------------------------
json 21.599k (± 0.6%) i/s (46.30 μs/i) - 108.579k in 5.027198s
```
Parsing:
```
== Parsing small nested array (121 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 47.497k i/100ms
oj 54.115k i/100ms
oj strict 53.854k i/100ms
Oj::Parser 150.904k i/100ms
rapidjson 80.775k i/100ms
Calculating -------------------------------------
json 481.096k (± 1.1%) i/s (2.08 μs/i) - 2.422M in 5.035657s
oj 554.878k (± 0.6%) i/s (1.80 μs/i) - 2.814M in 5.071521s
oj strict 547.888k (± 0.7%) i/s (1.83 μs/i) - 2.747M in 5.013212s
Oj::Parser 1.545M (± 0.4%) i/s (647.16 ns/i) - 7.847M in 5.078302s
rapidjson 822.422k (± 0.6%) i/s (1.22 μs/i) - 4.120M in 5.009178s
Comparison:
json: 481096.4 i/s
Oj::Parser: 1545223.5 i/s - 3.21x faster
rapidjson: 822422.4 i/s - 1.71x faster
oj: 554877.7 i/s - 1.15x faster
oj strict: 547887.7 i/s - 1.14x faster
== Parsing small hash (65 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 154.479k i/100ms
oj 220.283k i/100ms
oj strict 249.928k i/100ms
Oj::Parser 445.062k i/100ms
rapidjson 289.615k i/100ms
Calculating -------------------------------------
json 1.581M (± 3.0%) i/s (632.55 ns/i) - 8.033M in 5.086476s
oj 2.202M (± 3.5%) i/s (454.08 ns/i) - 11.014M in 5.008146s
oj strict 2.498M (± 3.5%) i/s (400.25 ns/i) - 12.496M in 5.008245s
Oj::Parser 4.640M (± 0.4%) i/s (215.50 ns/i) - 23.588M in 5.083443s
rapidjson 3.111M (± 0.3%) i/s (321.44 ns/i) - 15.639M in 5.027097s
Comparison:
json: 1580898.5 i/s
Oj::Parser: 4640298.1 i/s - 2.94x faster
rapidjson: 3111005.2 i/s - 1.97x faster
oj strict: 2498421.4 i/s - 1.58x faster
oj: 2202276.6 i/s - 1.39x faster
== Parsing test from oj (256 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 37.580k i/100ms
oj 41.899k i/100ms
oj strict 50.731k i/100ms
Oj::Parser 74.589k i/100ms
rapidjson 50.954k i/100ms
Calculating -------------------------------------
json 382.150k (± 1.0%) i/s (2.62 μs/i) - 1.917M in 5.015737s
oj 420.282k (± 0.2%) i/s (2.38 μs/i) - 2.137M in 5.084338s
oj strict 511.758k (± 0.5%) i/s (1.95 μs/i) - 2.587M in 5.055821s
Oj::Parser 759.087k (± 0.3%) i/s (1.32 μs/i) - 3.804M in 5.011388s
rapidjson 518.273k (± 1.8%) i/s (1.93 μs/i) - 2.599M in 5.015867s
Comparison:
json: 382149.6 i/s
Oj::Parser: 759087.1 i/s - 1.99x faster
rapidjson: 518272.8 i/s - 1.36x faster
oj strict: 511758.4 i/s - 1.34x faster
oj: 420282.5 i/s - 1.10x faster
== Parsing twitter.json (567916 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 52.000 i/100ms
oj 63.000 i/100ms
oj strict 74.000 i/100ms
Oj::Parser 79.000 i/100ms
rapidjson 56.000 i/100ms
Calculating -------------------------------------
json 522.896 (± 0.4%) i/s (1.91 ms/i) - 2.652k in 5.071809s
oj 624.849 (± 0.6%) i/s (1.60 ms/i) - 3.150k in 5.041398s
oj strict 737.779 (± 0.4%) i/s (1.36 ms/i) - 3.700k in 5.015117s
Oj::Parser 789.254 (± 0.3%) i/s (1.27 ms/i) - 3.950k in 5.004764s
rapidjson 565.663 (± 0.4%) i/s (1.77 ms/i) - 2.856k in 5.049015s
Comparison:
json: 522.9 i/s
Oj::Parser: 789.3 i/s - 1.51x faster
oj strict: 737.8 i/s - 1.41x faster
oj: 624.8 i/s - 1.19x faster
rapidjson: 565.7 i/s - 1.08x faster
== Parsing citm_catalog.json (1727030 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 27.000 i/100ms
oj 31.000 i/100ms
oj strict 36.000 i/100ms
Oj::Parser 42.000 i/100ms
rapidjson 38.000 i/100ms
Calculating -------------------------------------
json 305.248 (± 0.3%) i/s (3.28 ms/i) - 1.539k in 5.041813s
oj 320.265 (± 3.4%) i/s (3.12 ms/i) - 1.612k in 5.039715s
oj strict 373.701 (± 1.6%) i/s (2.68 ms/i) - 1.872k in 5.010633s
Oj::Parser 457.792 (± 0.4%) i/s (2.18 ms/i) - 2.310k in 5.046049s
rapidjson 350.933 (± 8.8%) i/s (2.85 ms/i) - 1.748k in 5.052491s
Comparison:
json: 305.2 i/s
Oj::Parser: 457.8 i/s - 1.50x faster
oj strict: 373.7 i/s - 1.22x faster
rapidjson: 350.9 i/s - 1.15x faster
oj: 320.3 i/s - 1.05x faster
== Parsing canada.json (2251051 bytes)
ruby 3.3.4 (2024-07-09 revision be1089c8ec) +YJIT [arm64-darwin23]
Warming up --------------------------------------
json 2.000 i/100ms
oj 2.000 i/100ms
oj strict 2.000 i/100ms
Oj::Parser 2.000 i/100ms
rapidjson 28.000 i/100ms
Calculating -------------------------------------
json 29.216 (± 6.8%) i/s (34.23 ms/i) - 146.000 in 5.053753s
oj 24.899 (± 0.0%) i/s (40.16 ms/i) - 126.000 in 5.061915s
oj strict 24.828 (± 4.0%) i/s (40.28 ms/i) - 124.000 in 5.003067s
Oj::Parser 30.867 (± 3.2%) i/s (32.40 ms/i) - 156.000 in 5.057104s
rapidjson 285.761 (± 1.0%) i/s (3.50 ms/i) - 1.456k in 5.095715s
Comparison:
json: 29.2 i/s
rapidjson: 285.8 i/s - 9.78x faster
Oj::Parser: 30.9 i/s - same-ish: difference falls within error
oj: 24.9 i/s - 1.17x slower
oj strict: 24.8 i/s - 1.18x slower
```
byroot
force-pushed
the
revamp-benchmarks
branch
from
e76f04e to
d826daa
Compare
eregon
reviewed
Oct 8, 2024
| @@ -1,41 +0,0 @@ | |||
| require 'benchmark/ips' | |||
There was a problem hiding this comment.
I'd like to keep this benchmark (added in #580) as it's very simple (e.g. it doesn't need to run under Bundler) and is a convenient way to compare the pure-Ruby backend and the extension.
Do you mind if I add it back under benchmark?
There was a problem hiding this comment.
Alright, I just moved it: 7b68800
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a large number of outstanding performance PRs that I want to merge, but we need a decent benchmark to judge if they are effective.
I went to borrow rapidjson's benchmark suite, which is a good start.
I only kept the comparison with Oj and RapidJSON, because YAJL is slower on most benchmarks, so little point comparing to it.
FYI @jhawthorn
Encoding
Parsing