Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kwxm/multi cost model benchmark test #6063

Closed
wants to merge 16 commits into from
Closed

Kwxm/multi cost model benchmark test #6063

wants to merge 16 commits into from

Conversation

kwxm
Copy link
Contributor

@kwxm kwxm commented May 20, 2024

DON'T REVIEW THIS OR MERGE IT!

There's an imminent PR to update the cost modelling machinery to handle multiple cost models. The default cost model used for testing/benchmarking is now the provisional Conway cost model, which updates the costs of many things. I want to check some stuff using the old cost model with the new machinery though, and that's what this is for.

@kwxm kwxm added Do not merge Don't look here yet EXPERIMENT Experiments that we probably don't want to merge No Changelog Required Add this to skip the Changelog Check labels May 20, 2024
@kwxm
Copy link
Contributor Author

kwxm commented May 20, 2024

/benchmark validation

@github-actions GitHub Actions
Copy link
Contributor

Click here to check the status of your benchmark.

@kwxm
Copy link
Contributor Author

kwxm commented May 20, 2024

/benchmark nofib

@kwxm
Copy link
Contributor Author

kwxm commented May 20, 2024

/benchmark bls12-381-benchmarks

@github-actions GitHub Actions
Copy link
Contributor

Comparing benchmark results of 'validation' on '5ee29364a' (base) and 'd0ca303ad' (PR)

Results table
Script 5ee2936 d0ca303 Change
auction_1-1 172.4 μs 174.0 μs +0.9%
auction_1-2 627.0 μs 628.7 μs +0.3%
auction_1-3 624.9 μs 622.0 μs -0.5%
auction_1-4 223.5 μs 226.2 μs +1.2%
auction_2-1 172.6 μs 175.0 μs +1.4%
auction_2-2 627.7 μs 626.7 μs -0.2%
auction_2-3 822.5 μs 819.0 μs -0.4%
auction_2-4 627.2 μs 624.6 μs -0.4%
auction_2-5 223.4 μs 226.2 μs +1.3%
crowdfunding-success-1 202.4 μs 204.5 μs +1.0%
crowdfunding-success-2 201.7 μs 205.0 μs +1.6%
crowdfunding-success-3 202.1 μs 203.9 μs +0.9%
currency-1 244.1 μs 245.0 μs +0.4%
escrow-redeem_1-1 338.4 μs 341.9 μs +1.0%
escrow-redeem_1-2 341.1 μs 344.8 μs +1.1%
escrow-redeem_2-1 392.2 μs 395.9 μs +0.9%
escrow-redeem_2-2 392.1 μs 396.9 μs +1.2%
escrow-redeem_2-3 393.1 μs 396.6 μs +0.9%
escrow-refund-1 150.1 μs 152.0 μs +1.3%
future-increase-margin-1 245.5 μs 245.4 μs -0.0%
future-increase-margin-2 527.2 μs 528.2 μs +0.2%
future-increase-margin-3 528.0 μs 529.2 μs +0.2%
future-increase-margin-4 480.8 μs 479.9 μs -0.2%
future-increase-margin-5 800.7 μs 803.4 μs +0.3%
future-pay-out-1 246.1 μs 244.5 μs -0.7%
future-pay-out-2 529.1 μs 527.8 μs -0.2%
future-pay-out-3 528.2 μs 528.4 μs +0.0%
future-pay-out-4 804.9 μs 805.7 μs +0.1%
future-settle-early-1 245.3 μs 245.5 μs +0.1%
future-settle-early-2 529.8 μs 530.3 μs +0.1%
future-settle-early-3 527.9 μs 528.6 μs +0.1%
future-settle-early-4 606.5 μs 607.1 μs +0.1%
game-sm-success_1-1 374.6 μs 377.1 μs +0.7%
game-sm-success_1-2 193.2 μs 193.6 μs +0.2%
game-sm-success_1-3 627.8 μs 629.2 μs +0.2%
game-sm-success_1-4 218.8 μs 219.3 μs +0.2%
game-sm-success_2-1 377.1 μs 378.1 μs +0.3%
game-sm-success_2-2 193.1 μs 193.7 μs +0.3%
game-sm-success_2-3 626.7 μs 627.3 μs +0.1%
game-sm-success_2-4 219.2 μs 219.5 μs +0.1%
game-sm-success_2-5 629.7 μs 627.9 μs -0.3%
game-sm-success_2-6 219.0 μs 221.0 μs +0.9%
multisig-sm-1 385.4 μs 384.8 μs -0.2%
multisig-sm-2 374.8 μs 374.3 μs -0.1%
multisig-sm-3 379.3 μs 378.6 μs -0.2%
multisig-sm-4 383.2 μs 381.6 μs -0.4%
multisig-sm-5 545.9 μs 541.9 μs -0.7%
multisig-sm-6 385.6 μs 384.0 μs -0.4%
multisig-sm-7 373.8 μs 374.0 μs +0.1%
multisig-sm-8 381.5 μs 380.6 μs -0.2%
multisig-sm-9 382.2 μs 381.7 μs -0.1%
multisig-sm-10 546.9 μs 542.5 μs -0.8%
ping-pong-1 319.6 μs 318.1 μs -0.5%
ping-pong-2 321.0 μs 319.0 μs -0.6%
ping-pong_2-1 194.0 μs 195.0 μs +0.5%
prism-1 163.1 μs 163.2 μs +0.1%
prism-2 400.3 μs 400.5 μs +0.0%
prism-3 358.7 μs 362.1 μs +0.9%
pubkey-1 137.0 μs 138.9 μs +1.4%
stablecoin_1-1 902.6 μs 900.3 μs -0.3%
stablecoin_1-2 189.3 μs 189.4 μs +0.1%
stablecoin_1-3 1.039 ms 1.033 ms -0.6%
stablecoin_1-4 199.8 μs 199.9 μs +0.1%
stablecoin_1-5 1.325 ms 1.314 ms -0.8%
stablecoin_1-6 246.4 μs 248.0 μs +0.6%
stablecoin_2-1 903.8 μs 898.8 μs -0.6%
stablecoin_2-2 188.8 μs 189.4 μs +0.3%
stablecoin_2-3 1.038 ms 1.028 ms -1.0%
stablecoin_2-4 199.3 μs 200.0 μs +0.4%
token-account-1 183.8 μs 184.9 μs +0.6%
token-account-2 325.3 μs 326.7 μs +0.4%
uniswap-1 405.3 μs 404.9 μs -0.1%
uniswap-2 213.9 μs 214.2 μs +0.1%
uniswap-3 1.708 ms 1.707 ms -0.1%
uniswap-4 324.6 μs 326.5 μs +0.6%
uniswap-5 1.121 ms 1.130 ms +0.8%
uniswap-6 311.4 μs 312.1 μs +0.2%
vesting-1 337.9 μs 338.3 μs +0.1%
TOTAL 34.45 ms 34.47 ms +0.0%

@github-actions GitHub Actions
Copy link
Contributor

Click here to check the status of your benchmark.

@github-actions GitHub Actions
Copy link
Contributor

Comparing benchmark results of 'nofib' on '5ee29364a' (base) and 'd0ca303ad' (PR)

Results table
Script 5ee2936 d0ca303 Change
clausify/formula1 2.935 ms 2.941 ms +0.2%
clausify/formula2 3.917 ms 3.929 ms +0.3%
clausify/formula3 10.71 ms 10.75 ms +0.4%
clausify/formula4 24.41 ms 24.51 ms +0.4%
clausify/formula5 52.08 ms 52.01 ms -0.1%
knights/4x4 16.83 ms 16.85 ms +0.1%
knights/6x6 44.34 ms 44.19 ms -0.3%
knights/8x8 77.79 ms 77.84 ms +0.1%
primetest/05digits 9.832 ms 9.773 ms -0.6%
primetest/08digits 15.88 ms 15.82 ms -0.4%
primetest/10digits 19.07 ms 19.01 ms -0.3%
primetest/20digits 37.78 ms 37.67 ms -0.3%
primetest/30digits 58.13 ms 57.72 ms -0.7%
primetest/40digits 78.92 ms 78.08 ms -1.1%
primetest/50digits 95.91 ms 95.16 ms -0.8%
queens4x4/bt 5.122 ms 5.137 ms +0.3%
queens4x4/bm 6.521 ms 6.585 ms +1.0%
queens4x4/bjbt1 6.272 ms 6.286 ms +0.2%
queens4x4/bjbt2 5.871 ms 5.868 ms -0.1%
queens4x4/fc 13.27 ms 13.38 ms +0.8%
queens5x5/bt 69.64 ms 69.57 ms -0.1%
queens5x5/bm 73.16 ms 73.94 ms +1.1%
queens5x5/bjbt1 81.48 ms 81.40 ms -0.1%
queens5x5/bjbt2 79.10 ms 79.04 ms -0.1%
queens5x5/fc 167.8 ms 169.5 ms +1.0%
TOTAL 1.057 s 1.057 s +0.0%

@github-actions GitHub Actions
Copy link
Contributor

Click here to check the status of your benchmark.

@github-actions GitHub Actions
Copy link
Contributor

Comparing benchmark results of 'bls12-381-benchmarks' on '5ee29364a' (base) and 'd0ca303ad' (PR)

Results table
Script 5ee2936 d0ca303 Change
hashAndAddG1/0 40.87 μs 40.77 μs -0.2%
hashAndAddG1/10 564.4 μs 563.7 μs -0.1%
hashAndAddG1/20 1.088 ms 1.090 ms +0.2%
hashAndAddG1/30 1.616 ms 1.614 ms -0.1%
hashAndAddG1/40 2.137 ms 2.133 ms -0.2%
hashAndAddG1/50 2.660 ms 2.659 ms -0.0%
hashAndAddG1/60 3.185 ms 3.181 ms -0.1%
hashAndAddG1/70 3.710 ms 3.710 ms 0.0%
hashAndAddG1/80 4.232 ms 4.223 ms -0.2%
hashAndAddG1/90 4.756 ms 4.750 ms -0.1%
hashAndAddG1/100 5.287 ms 5.270 ms -0.3%
hashAndAddG1/110 5.803 ms 5.811 ms +0.1%
hashAndAddG1/120 6.330 ms 6.319 ms -0.2%
hashAndAddG1/130 6.853 ms 6.846 ms -0.1%
hashAndAddG1/140 7.375 ms 7.365 ms -0.1%
hashAndAddG1/150 7.898 ms 7.892 ms -0.1%
hashAndAddG2/0 50.28 μs 50.59 μs +0.6%
hashAndAddG2/10 1.719 ms 1.719 ms 0.0%
hashAndAddG2/20 3.388 ms 3.388 ms 0.0%
hashAndAddG2/30 5.055 ms 5.055 ms 0.0%
hashAndAddG2/40 6.722 ms 6.723 ms +0.0%
hashAndAddG2/50 8.390 ms 8.390 ms 0.0%
hashAndAddG2/60 10.06 ms 10.06 ms 0.0%
hashAndAddG2/70 11.72 ms 11.72 ms 0.0%
hashAndAddG2/80 13.39 ms 13.39 ms 0.0%
hashAndAddG2/90 15.06 ms 15.06 ms 0.0%
hashAndAddG2/100 16.72 ms 16.73 ms +0.1%
hashAndAddG2/110 18.39 ms 18.39 ms 0.0%
hashAndAddG2/120 20.05 ms 20.06 ms +0.0%
hashAndAddG2/130 21.73 ms 21.73 ms 0.0%
hashAndAddG2/140 23.40 ms 23.40 ms 0.0%
hashAndAddG2/150 25.06 ms 25.07 ms +0.0%
uncompressAndAddG1/0 40.83 μs 40.77 μs -0.1%
uncompressAndAddG1/10 580.7 μs 578.8 μs -0.3%
uncompressAndAddG1/20 1.120 ms 1.111 ms -0.8%
uncompressAndAddG1/30 1.660 ms 1.645 ms -0.9%
uncompressAndAddG1/40 2.198 ms 2.191 ms -0.3%
uncompressAndAddG1/50 2.739 ms 2.720 ms -0.7%
uncompressAndAddG1/60 3.279 ms 3.263 ms -0.5%
uncompressAndAddG1/70 3.811 ms 3.796 ms -0.4%
uncompressAndAddG1/80 4.353 ms 4.319 ms -0.8%
uncompressAndAddG1/90 4.896 ms 4.860 ms -0.7%
uncompressAndAddG1/100 5.432 ms 5.413 ms -0.3%
uncompressAndAddG1/110 5.973 ms 5.929 ms -0.7%
uncompressAndAddG1/120 6.515 ms 6.454 ms -0.9%
uncompressAndAddG1/130 7.052 ms 7.038 ms -0.2%
uncompressAndAddG1/140 7.591 ms 7.553 ms -0.5%
uncompressAndAddG1/150 8.131 ms 8.090 ms -0.5%
uncompressAndAddG2/0 50.28 μs 50.57 μs +0.6%
uncompressAndAddG2/10 815.3 μs 816.2 μs +0.1%
uncompressAndAddG2/20 1.580 ms 1.581 ms +0.1%
uncompressAndAddG2/30 2.345 ms 2.346 ms +0.0%
uncompressAndAddG2/40 3.110 ms 3.112 ms +0.1%
uncompressAndAddG2/50 3.875 ms 3.877 ms +0.1%
uncompressAndAddG2/60 4.640 ms 4.643 ms +0.1%
uncompressAndAddG2/70 5.406 ms 5.409 ms +0.1%
uncompressAndAddG2/80 6.171 ms 6.172 ms +0.0%
uncompressAndAddG2/90 6.936 ms 6.940 ms +0.1%
uncompressAndAddG2/100 7.699 ms 7.704 ms +0.1%
uncompressAndAddG2/110 8.467 ms 8.470 ms +0.0%
uncompressAndAddG2/120 9.230 ms 9.236 ms +0.1%
uncompressAndAddG2/130 9.995 ms 9.999 ms +0.0%
uncompressAndAddG2/140 10.75 ms 10.77 ms +0.2%
uncompressAndAddG2/150 11.52 ms 11.53 ms +0.1%
pairing 1.080 ms 1.079 ms -0.1%
groth16Verify 1.967 ms 1.963 ms -0.2%
simpleVerify 1.287 ms 1.287 ms 0.0%
vrf 1.212 ms 1.212 ms 0.0%
g1Verify 1.173 ms 1.173 ms 0.0%
g2Verify 1.081 ms 1.081 ms 0.0%
aggregateSignatureSingleKey 2.684 ms 2.683 ms -0.0%
aggregateSignatureMultiKey 3.428 ms 3.428 ms 0.0%
schnorrG1Verify 318.0 μs 316.8 μs -0.4%
schnorrG2Verify 550.9 μs 550.7 μs -0.0%
TOTAL 437.2 ms 436.8 ms -0.1%

@kwxm
Copy link
Contributor Author

kwxm commented May 20, 2024

OK, that looks good.

@kwxm kwxm closed this May 20, 2024
@kwxm kwxm deleted the kwxm/multi-cost-model-benchmark-test branch May 20, 2024 17:05
@kwxm kwxm mentioned this pull request May 23, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
Do not merge Don't look here yet EXPERIMENT Experiments that we probably don't want to merge No Changelog Required Add this to skip the Changelog Check
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kwxm