Add benchmark script and diagnosis

[ohbarye]

Change

This pull request adds a benchmark script to measure which concurrency method is the best for each scenario.

• simple success: No worker is the bast.
• CPU-bound: Ractor is the best.
• IO-bound: No worker is the best.
  • 💭 ...really?
• simple failure case with shrink: No worker is the best.

Based on the benchmark's result, I changed the default worker to :none since it's the happiest way for users.

Benchmarks

The following benchmarks are the results of running the benchmark suite.

• macOS 13.3.1, Apple M1 Pro 10 cores (8 performance and 2 efficiency)
• ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
• pbt commit hash 6582b27

---

Benchmark success:simple

This runs a script that does not do any IO or CPU bound work.

ruby benchmark/success_simple.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
Warming up --------------------------------------
ractor    20.000 i/100ms
process     3.000 i/100ms
thread   126.000 i/100ms
none   668.000 i/100ms
Calculating -------------------------------------
ractor    173.918 (±11.5%) i/s -    880.000 in   5.129007s
process     28.861 (± 3.5%) i/s -    147.000 in   5.100393s
thread      1.130k (± 5.5%) i/s -      5.670k in   5.031552s
none      6.534k (± 2.3%) i/s -     32.732k in   5.011885s

Benchmark success:cpu_bound

This runs a script that does CPU bound work.

ruby benchmark/success_cpu_bound.rb
Call tarai function with(9, 4, 0)

ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
Warming up --------------------------------------
ractor     3.000 i/100ms
process     2.000 i/100ms
thread     1.000 i/100ms
none     1.000 i/100ms
Calculating -------------------------------------
ractor     32.788 (± 6.1%) i/s -    165.000 in   5.057492s
process     22.098 (± 4.5%) i/s -    112.000 in   5.080410s
thread      7.439 (± 0.0%) i/s -     38.000 in   5.108195s
none      7.494 (± 0.0%) i/s -     38.000 in   5.070547s

Benchmark success:io_bound

This runs a script that does IO bound work.

ruby benchmark/success_io_bound.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
Warming up --------------------------------------
ractor    11.000 i/100ms
process     3.000 i/100ms
thread    17.000 i/100ms
none    22.000 i/100ms
Calculating -------------------------------------
ractor     82.488 (±14.5%) i/s -    407.000 in   5.054559s
process     35.403 (± 5.6%) i/s -    177.000 in   5.013818s
thread    143.022 (± 7.7%) i/s -    714.000 in   5.021129s
none    223.252 (± 9.0%) i/s -      1.122k in   5.071176s

Benchmark failure:simple

This runs a script that fails and shrink happens.

ruby benchmark/failure_simple.rb
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin22]
Warming up --------------------------------------
ractor     6.000 i/100ms
process     1.000 i/100ms
thread     9.000 i/100ms
none   815.000 i/100ms
Calculating -------------------------------------
ractor     62.770 (±15.9%) i/s -    306.000 in   5.009858s
process      1.783 (± 0.0%) i/s -      9.000 in   5.049606s
thread     85.218 (± 9.4%) i/s -    423.000 in   5.007178s
none      5.387k (± 3.3%) i/s -     27.710k in   5.149867s