Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chained benchmarks #30

Merged
merged 17 commits into from
Jan 9, 2024
Merged

Chained benchmarks #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
683d354
Chained benchmarks
adamw Jan 8, 2024
fbad9a0
Fix benchmark run times on CI
adamw Jan 8, 2024
24869a1
Fix benchmark run times on CI
adamw Jan 8, 2024
ab280da
Don't run chained tests on CI
adamw Jan 8, 2024
699d5b4
Fix build
adamw Jan 8, 2024
a8b1b8e
Try decreasing spins to 100
adamw Jan 8, 2024
acee4ee
Try running chained benchmarks on CI
adamw Jan 8, 2024
b162712
Try setting spins to 1000
adamw Jan 8, 2024
2729f9a
Check exchanger algorithm
adamw Jan 9, 2024
e806c4a
Try smaller number of loops of systems with a small number of CPUs
adamw Jan 9, 2024
56eb643
Try conditionally yielding
adamw Jan 9, 2024
4171b47
Try 1000/100 iterations
adamw Jan 9, 2024
3ed73b6
Adjust spins
adamw Jan 9, 2024
0ae1713
Simplify benchmarks
adamw Jan 9, 2024
028a137
Rename tests
adamw Jan 9, 2024
a273f71
Add java chained benchmarks
adamw Jan 9, 2024
635acf5
Readme
adamw Jan 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,21 +42,21 @@ jobs:
id: benchmark_java
run: |
mvn package -DskipTests=true
java -Djmh.executor=VIRTUAL -jar bench/bench-java/target/benchmarks.jar -i 5 -wi 5 -f 3 -to 2100ms -r 2000ms -w 2000ms -rf json | tee out.txt
java -Djmh.executor=VIRTUAL -jar bench/bench-java/target/benchmarks.jar -i 5 -wi 5 -f 1 -to 1100ms -r 1000ms -w 1000ms -rf json -rff jmh-result-java.json | tee out.txt

echo 'output<<EOF' >> $GITHUB_OUTPUT
cat out.txt >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
- name: Run kotlin benchmarks
id: benchmark_kotlin
run: |
java -jar bench/bench-kotlin/target/benchmarks.jar -i 5 -wi 5 -f 3 -to 2100ms -r 2000ms -w 2000ms -rf json -rff jmh-result-kotlin.json | tee out.txt
java -jar bench/bench-kotlin/target/benchmarks.jar -i 5 -wi 5 -f 1 -to 1100ms -r 1000ms -w 1000ms -rf json -rff jmh-result-kotlin.json | tee out.txt

echo 'output<<EOF' >> $GITHUB_OUTPUT
cat out.txt >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
- name: Merge java and kotlin benchmark results
run: jq -s '.[0] + .[1]' jmh-result.json jmh-result-kotlin.json > jmh-result-both.json
run: jq -s '.[0] + .[1]' jmh-result-java.json jmh-result-kotlin.json > jmh-result-all.json
- name: Extract branch name
shell: bash
run: echo "branch=${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}" >> $GITHUB_OUTPUT
Expand All @@ -67,7 +67,7 @@ jobs:
if: steps.extract_branch.outputs.branch == 'main'
with:
tool: 'jmh'
output-file-path: jmh-result-both.json
output-file-path: jmh-result-all.json
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: true
# Show alert with commit comment on detecting possible performance regression
Expand Down
108 changes: 62 additions & 46 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,54 +214,70 @@ class Demo6 {
The project includes benchmarks implemented using JMH - both for the `Channel`, as well as for some built-in Java
synchronisation primitives (queues), as well as the Kotlin channel implementation.

The test results for version 0.0.1, run on an M1 Max MacBook Pro, with Java 21.0.1, are as follows:
The test results for version 0.0.4, run on an M1 Max MacBook Pro, with Java 21.0.1, are as follows:

```
Benchmark (capacity) Mode Cnt Score Error Units

// jox
RendezvousBenchmark.channel N/A avgt 30 176.499 ± 14.964 ns/op
RendezvousBenchmark.channel:receiveFromChannel N/A avgt 30 176.499 ± 14.964 ns/op
RendezvousBenchmark.channel:sendToChannel N/A avgt 30 176.499 ± 14.964 ns/op
RendezvousBenchmark.channel_iterative N/A avgt 30 209.041 ± 30.397 ns/op

BufferedBenchmark.channel 1 avgt 30 177.547 ± 14.626 ns/op
BufferedBenchmark.channel:receiveFromChannel 1 avgt 30 177.547 ± 14.626 ns/op
BufferedBenchmark.channel:sendToChannel 1 avgt 30 177.547 ± 14.626 ns/op
BufferedBenchmark.channel 10 avgt 30 135.838 ± 14.578 ns/op
BufferedBenchmark.channel:receiveFromChannel 10 avgt 30 135.838 ± 14.578 ns/op
BufferedBenchmark.channel:sendToChannel 10 avgt 30 135.838 ± 14.578 ns/op
BufferedBenchmark.channel 100 avgt 30 92.837 ± 13.936 ns/op
BufferedBenchmark.channel:receiveFromChannel 100 avgt 30 92.837 ± 13.936 ns/op
BufferedBenchmark.channel:sendToChannel 100 avgt 30 92.836 ± 13.935 ns/op
BufferedBenchmark.channel_iterative 1 avgt 30 185.138 ± 14.382 ns/op
BufferedBenchmark.channel_iterative 10 avgt 30 126.594 ± 12.089 ns/op
BufferedBenchmark.channel_iterative 100 avgt 30 83.534 ± 6.540 ns/op

// java
RendezvousBenchmark.exchanger N/A avgt 30 177.630 ± 152.388 ns/op
RendezvousBenchmark.exchanger:exchange1 N/A avgt 30 177.630 ± 152.388 ns/op
RendezvousBenchmark.exchanger:exchange2 N/A avgt 30 177.630 ± 152.388 ns/op
RendezvousBenchmark.synchronous_queue N/A avgt 30 978.826 ± 188.831 ns/op
RendezvousBenchmark.synchronous_queue:putToSynchronousQueue N/A avgt 30 978.826 ± 188.830 ns/op
RendezvousBenchmark.synchronous_queue:takeFromSynchronousQueue N/A avgt 30 978.825 ± 188.832 ns/op

BufferedBenchmark.array_blocking_queue 1 avgt 30 2266.799 ± 231.198 ns/op
BufferedBenchmark.array_blocking_queue:putToArrayBlockingQueue 1 avgt 30 2266.798 ± 231.197 ns/op
BufferedBenchmark.array_blocking_queue:takeFromArrayBlockingQueue 1 avgt 30 2266.799 ± 231.199 ns/op
BufferedBenchmark.array_blocking_queue 10 avgt 30 450.796 ± 93.496 ns/op
BufferedBenchmark.array_blocking_queue:putToArrayBlockingQueue 10 avgt 30 450.795 ± 93.495 ns/op
BufferedBenchmark.array_blocking_queue:takeFromArrayBlockingQueue 10 avgt 30 450.797 ± 93.497 ns/op
BufferedBenchmark.array_blocking_queue 100 avgt 30 147.962 ± 9.743 ns/op
BufferedBenchmark.array_blocking_queue:putToArrayBlockingQueue 100 avgt 30 147.962 ± 9.743 ns/op
BufferedBenchmark.array_blocking_queue:takeFromArrayBlockingQueue 100 avgt 30 147.962 ± 9.743 ns/op

// kotlin
RendezvousKotlinBenchmark.sendReceiveUsingDefaultDispatcher N/A avgt 30 108.338 ± 0.538 ns/op

BufferedKotlinBenchmark.sendReceiveUsingDefaultDispatcher 1 avgt 30 86.614 ± 0.784 ns/op
BufferedKotlinBenchmark.sendReceiveUsingDefaultDispatcher 10 avgt 30 40.153 ± 0.221 ns/op
BufferedKotlinBenchmark.sendReceiveUsingDefaultDispatcher 100 avgt 30 26.764 ± 0.022 ns/op
Benchmark (capacity) (chainLength) Mode Cnt Score Error Units

// jox - single channel
BufferedBenchmark.channel 1 N/A avgt 20 210.366 ± 18.979 ns/op
BufferedBenchmark.channel 10 N/A avgt 20 148.691 ± 25.368 ns/op
BufferedBenchmark.channel 100 N/A avgt 20 149.499 ± 22.495 ns/op

RendezvousBenchmark.channel N/A N/A avgt 20 187.940 ± 8.783 ns/op

// kotlin - single channel
BufferedKotlinBenchmark.channel_defaultDispatcher 1 N/A avgt 30 85.027 ± 0.709 ns/op
BufferedKotlinBenchmark.channel_defaultDispatcher 10 N/A avgt 30 40.095 ± 0.452 ns/op
BufferedKotlinBenchmark.channel_defaultDispatcher 100 N/A avgt 30 26.879 ± 0.063 ns/op

RendezvousKotlinBenchmark.channel_defaultDispatcher N/A N/A avgt 30 116.664 ± 10.099 ns/op

// jox - selects
SelectBenchmark.selectWithSingleClause N/A N/A avgt 20 353.074 ± 27.860 ns/op
SelectBenchmark.selectWithTwoClauses N/A N/A avgt 20 651.050 ± 31.037 ns/op

// kotlin - selects
SelectKotlinBenchmark.selectWithSingleClause_defaultDispatcher N/A N/A avgt 30 169.823 ± 1.250 ns/op
SelectKotlinBenchmark.selectWithTwoClauses_defaultDispatcher N/A N/A avgt 30 227.413 ± 2.659 ns/op

// java built-in - single queue
BufferedBenchmark.arrayBlockingQueue 1 N/A avgt 20 2447.455 ± 427.354 ns/op
BufferedBenchmark.arrayBlockingQueue 10 N/A avgt 20 546.227 ± 96.690 ns/op
BufferedBenchmark.arrayBlockingQueue 100 N/A avgt 20 125.287 ± 4.387 ns/op

RendezvousBenchmark.exchanger N/A N/A avgt 20 106.114 ± 20.360 ns/op
RendezvousBenchmark.synchronousQueue N/A N/A avgt 20 869.988 ± 101.291 ns/op

// jox - multi channel
ChainedBenchmark.channelChain 0 100 avgt 20 225.370 ± 4.693 ns/op
ChainedBenchmark.channelChain 0 1000 avgt 20 173.997 ± 4.160 ns/op
ChainedBenchmark.channelChain 0 10000 avgt 20 160.097 ± 4.520 ns/op
ChainedBenchmark.channelChain 100 100 avgt 20 8.377 ± 0.133 ns/op
ChainedBenchmark.channelChain 100 1000 avgt 20 6.147 ± 0.054 ns/op
ChainedBenchmark.channelChain 100 10000 avgt 20 7.942 ± 0.447 ns/op

// kotlin - multi channel
ChainedKotlinBenchmark.channelChain_defaultDispatcher 0 100 avgt 30 96.106 ± 1.247 ns/op
ChainedKotlinBenchmark.channelChain_defaultDispatcher 0 1000 avgt 30 74.858 ± 0.810 ns/op
ChainedKotlinBenchmark.channelChain_defaultDispatcher 0 10000 avgt 30 72.894 ± 0.787 ns/op
ChainedKotlinBenchmark.channelChain_defaultDispatcher 100 100 avgt 30 5.164 ± 0.104 ns/op
ChainedKotlinBenchmark.channelChain_defaultDispatcher 100 1000 avgt 30 4.157 ± 0.029 ns/op
ChainedKotlinBenchmark.channelChain_defaultDispatcher 100 10000 avgt 30 4.965 ± 0.043 ns/op
ChainedKotlinBenchmark.channelChain_eventLoop 0 100 avgt 30 70.484 ± 0.431 ns/op
ChainedKotlinBenchmark.channelChain_eventLoop 0 1000 avgt 30 98.400 ± 1.003 ns/op
ChainedKotlinBenchmark.channelChain_eventLoop 0 10000 avgt 30 92.579 ± 1.650 ns/op
ChainedKotlinBenchmark.channelChain_eventLoop 100 100 avgt 30 27.052 ± 0.121 ns/op
ChainedKotlinBenchmark.channelChain_eventLoop 100 1000 avgt 30 25.982 ± 0.111 ns/op
ChainedKotlinBenchmark.channelChain_eventLoop 100 10000 avgt 30 27.276 ± 0.316 ns/op

// java built-in - multi queues
ChainedBenchmark.queueChain 0 100 avgt 20 186.677 ± 2.564 ns/op
ChainedBenchmark.queueChain 0 1000 avgt 20 108.954 ± 13.825 ns/op
ChainedBenchmark.queueChain 0 10000 avgt 20 101.643 ± 10.526 ns/op
ChainedBenchmark.queueChain 100 100 avgt 20 7.933 ± 0.546 ns/op
ChainedBenchmark.queueChain 100 1000 avgt 20 5.281 ± 0.261 ns/op
ChainedBenchmark.queueChain 100 10000 avgt 20 5.798 ± 0.058 ns/op
```

## Feedback
Expand Down
87 changes: 35 additions & 52 deletions bench/bench-java/src/main/java/com/softwaremill/jox/BufferedBenchmark.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,75 +8,58 @@
/**
* Buffered tests for {@link ArrayBlockingQueue} and {@link Channel}.
*/
@Warmup(iterations = 3, time = 5000, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 10, time = 5000, timeUnit = TimeUnit.MILLISECONDS)
// after the measurement time, we want to interrupt any pending methods (which might block, waiting for a partner)
// this needs to be slightly larger than the test time to avoid warnings
@Timeout(time = 5100, timeUnit = TimeUnit.MILLISECONDS)
@Fork(value = 3)
@Warmup(iterations = 3, time = 3000, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 10, time = 3000, timeUnit = TimeUnit.MILLISECONDS)
@Fork(value = 2)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Group)
@State(Scope.Benchmark)
public class BufferedBenchmark {
@Param({"1", "10", "100"})
public int capacity;

private ArrayBlockingQueue<Integer> queue;
private Channel<Integer> channel;
// going against jmh's best practises, the benchmarks are "iterative" (not using groups), for two reasons:
// (1) direct comparison w/ Kotlin, as we can't write a group-based benchmark there, due to suspended functions
// (2) the more complex benchmarks (which use higher numbers of threads) need to be enclosed in a single method anyway

@Setup
public void create() {
queue = new ArrayBlockingQueue<>(capacity);
channel = new Channel<>(capacity);
}

//

@Benchmark
@Group("array_blocking_queue")
@GroupThreads(1)
public void putToArrayBlockingQueue() throws InterruptedException {
queue.put(63);
}
private final static int OPERATIONS_PER_INVOCATION = 1_000_000;

@Benchmark
@Group("array_blocking_queue")
@GroupThreads(1)
public void takeFromArrayBlockingQueue() throws InterruptedException {
queue.take();
}

//
@OperationsPerInvocation(OPERATIONS_PER_INVOCATION)
public void arrayBlockingQueue() throws InterruptedException {
var queue = new ArrayBlockingQueue<>(capacity);
var t1 = Thread.startVirtualThread(() -> {
for (int i = 0; i < OPERATIONS_PER_INVOCATION; i++) {
try {
queue.put(63);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
});

@Benchmark
@Group("channel")
@GroupThreads(1)
public void sendToChannel() throws InterruptedException {
channel.send(63);
}
var t2 = Thread.startVirtualThread(() -> {
for (int i = 0; i < OPERATIONS_PER_INVOCATION; i++) {
try {
queue.take();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
});

@Benchmark
@Group("channel")
@GroupThreads(1)
public void receiveFromChannel() throws InterruptedException {
channel.receive();
t1.join();
t2.join();
}

//

// including an iterative benchmark, for direct comparison w/ Kotlin, as we can't write a group-based benchmark
// there, due to suspended functions

private final static int OPERATIONS_PER_INVOCATION = 1_000_000;

@Benchmark
@OperationsPerInvocation(OPERATIONS_PER_INVOCATION)
@Group("channel_iterative")
public void sendReceive() throws InterruptedException {
public void channel() throws InterruptedException {
var ch = new Channel<>(capacity);
var t1 = Thread.startVirtualThread(() -> {
for (int i = 0; i < OPERATIONS_PER_INVOCATION; i++) {
try {
channel.send(63);
ch.send(63);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
Expand All @@ -86,7 +69,7 @@ public void sendReceive() throws InterruptedException {
var t2 = Thread.startVirtualThread(() -> {
for (int i = 0; i < OPERATIONS_PER_INVOCATION; i++) {
try {
channel.receive();
ch.receive();
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
Expand Down
Loading
Loading