Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

disable the new pipeline for forge #15742

Merged
merged 1 commit into from
Jan 17, 2025
Merged

Conversation

msmouse
Copy link
Contributor

@msmouse msmouse commented Jan 15, 2025

Description

There's known edge case where fullnodes in CO mode would crash on epoch changes with the new pipeline.

How Has This Been Tested?

forge land blocking

Key Areas to Review

Type of Change

  • Tests

Which Components or Systems Does This Change Impact?

  • Developer Infrastructure - forge

Copy link

trunk-io bot commented Jan 15, 2025

⏱️ 4h 16m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
forge-realistic-env-graceful-workload-sweep / forge 45m 🟥
forge-realistic-env-workload-sweep / forge 40m 🟥
forge-realistic-env-load-sweep / forge 40m 🟩
rust-smoke-tests 31m 🟩
forge-realistic-env-graceful-overload / forge 30m 🟩
test-target-determinator 14m 🟩🟩🟩
check-dynamic-deps 10m 🟩🟩🟩🟩🟩
execution-performance / test-target-determinator 10m 🟩🟩
forge-framework-upgrade-test / forge 6m 🟥
rust-doc-tests 6m 🟩
rust-doc-tests 6m 🟩
rust-cargo-deny 5m 🟩🟩🟩
determine-test-metadata 4m 🟩🟩
fetch-last-released-docker-image-tag 3m 🟩🟩
semgrep/ci 2m 🟩🟩🟩🟩🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
execution-performance / single-node-performance 9s 15m -99%

settingsfeedbackdocs ⋅ learn more about trunk.io

@msmouse msmouse added the CICD:build-images when this label is present github actions will start build+push rust images from the PR. label Jan 15, 2025
@msmouse msmouse force-pushed the 0115-alden-disable-new-pipeline branch from afdf922 to 42df0fb Compare January 16, 2025 21:24
@msmouse msmouse enabled auto-merge (squash) January 16, 2025 21:24

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

@msmouse msmouse force-pushed the 0115-alden-disable-new-pipeline branch from 42df0fb to 4e56010 Compare January 17, 2025 00:05

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite compat success on 6593fb81261f25490ffddc2252a861c994234c2a ==> 4e560107406937b11e70d006d527f743fb57c67b

Compatibility test results for 6593fb81261f25490ffddc2252a861c994234c2a ==> 4e560107406937b11e70d006d527f743fb57c67b (PR)
1. Check liveness of validators at old version: 6593fb81261f25490ffddc2252a861c994234c2a
compatibility::simple-validator-upgrade::liveness-check : committed: 17704.18 txn/s, latency: 1983.79 ms, (p50: 2100 ms, p70: 2100, p90: 2200 ms, p99: 2400 ms), latency samples: 567540
2. Upgrading first Validator to new version: 4e560107406937b11e70d006d527f743fb57c67b
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 5009.50 txn/s, latency: 6150.45 ms, (p50: 6900 ms, p70: 7400, p90: 7600 ms, p99: 7700 ms), latency samples: 99360
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 4957.58 txn/s, latency: 6966.44 ms, (p50: 7700 ms, p70: 7700, p90: 7900 ms, p99: 8100 ms), latency samples: 175720
3. Upgrading rest of first batch to new version: 4e560107406937b11e70d006d527f743fb57c67b
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 4541.99 txn/s, latency: 6826.88 ms, (p50: 7600 ms, p70: 8200, p90: 8500 ms, p99: 8600 ms), latency samples: 94260
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 4360.75 txn/s, latency: 7680.14 ms, (p50: 8400 ms, p70: 8600, p90: 8800 ms, p99: 9000 ms), latency samples: 157020
4. upgrading second batch to new version: 4e560107406937b11e70d006d527f743fb57c67b
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 8753.55 txn/s, latency: 3425.81 ms, (p50: 3900 ms, p70: 4200, p90: 4500 ms, p99: 4800 ms), latency samples: 160380
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8955.04 txn/s, latency: 3813.77 ms, (p50: 4100 ms, p70: 4300, p90: 4500 ms, p99: 4800 ms), latency samples: 294860
5. check swarm health
Compatibility test for 6593fb81261f25490ffddc2252a861c994234c2a ==> 4e560107406937b11e70d006d527f743fb57c67b passed
Test Ok

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 4e560107406937b11e70d006d527f743fb57c67b

two traffics test: inner traffic : committed: 14305.40 txn/s, latency: 2775.32 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3900 ms), latency samples: 5439220
two traffics test : committed: 99.98 txn/s, latency: 1488.03 ms, (p50: 1400 ms, p70: 1500, p90: 1600 ms, p99: 2800 ms), latency samples: 1800
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 1.703, avg: 1.537", "ConsensusProposalToOrdered: max: 0.301, avg: 0.294", "ConsensusOrderedToCommit: max: 0.406, avg: 0.386", "ConsensusProposalToCommit: max: 0.707, avg: 0.680"]
Max non-epoch-change gap was: 1 rounds at version 3390267 (avg 0.00) [limit 4], 2.38s no progress at version 3390267 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.66s no progress at version 1617281 (avg 0.66s) [limit 16].
Test Ok

@msmouse msmouse merged commit 8c11da3 into main Jan 17, 2025
43 of 46 checks passed
@msmouse msmouse deleted the 0115-alden-disable-new-pipeline branch January 17, 2025 17:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CICD:build-images when this label is present github actions will start build+push rust images from the PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants