tests: Add retry if promote member fails. #19272
Conversation
|
Hi @jmao-dd. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted filessee 29 files with indirect coverage changes @@ Coverage Diff @@
## main #19272 +/- ##
==========================================
+ Coverage 68.81% 68.86% +0.04%
==========================================
Files 420 420
Lines 35649 35680 +31
==========================================
+ Hits 24533 24572 +39
+ Misses 9691 9684 -7
+ Partials 1425 1424 -1 Continue to review full report in Codecov by Sentry.
|
| "go.etcd.io/etcd/server/v3/etcdserver" | ||
|
|
| _, err = epc.Etcdctl(e2e.WithEndpoints(endpoints)).MemberPromote(ctx, id) | ||
| attempt := 1 | ||
| for err != nil && (attempt < 4) && strings.Contains(err.Error(), "can only promote a learner member which is in sync with leader") { | ||
| t.Logf("Learner is not ready yet, retry for the %v time", attempt) | ||
| time.Sleep(100 * time.Duration(attempt) * time.Millisecond) | ||
| _, err = epc.Etcdctl(e2e.WithEndpoints(endpoints)).MemberPromote(ctx, id) | ||
| attempt++ | ||
| } |
There was a problem hiding this comment.
| _, err = epc.Etcdctl(e2e.WithEndpoints(endpoints)).MemberPromote(ctx, id) | |
| attempt := 1 | |
| for err != nil && (attempt < 4) && strings.Contains(err.Error(), "can only promote a learner member which is in sync with leader") { | |
| t.Logf("Learner is not ready yet, retry for the %v time", attempt) | |
| time.Sleep(100 * time.Duration(attempt) * time.Millisecond) | |
| _, err = epc.Etcdctl(e2e.WithEndpoints(endpoints)).MemberPromote(ctx, id) | |
| attempt++ | |
| } | |
| attempt := 0 | |
| for attempt < 3 { | |
| _, err = epc.Etcdctl(e2e.WithEndpoints(endpoints)).MemberPromote(ctx, id) | |
| if err == nil || !strings.Contains(err.Error(), "can only promote a learner member which is in sync with leader"){ | |
| break | |
| } | |
| time.Sleep(100 * time.Millisecond) | |
| attempt++ | |
| } |
There was a problem hiding this comment.
You did not resolve this comment correctly. We only need to call MemberPromote inside the loop. It causes a regression #19279
There was a problem hiding this comment.
oops, sorry I didn't notice the double call to MemberPromote.
|
Please squash the commit. |
To handle the case of learner not being ready when being promoted, add retry with some delay to improve the stability of e2e tests. Signed-off-by: Jiayin Mao <[email protected]>
jmao-dd
force-pushed
the
jmao/19216-flakey-promote-learner-test
branch
from
5a868a8 to
692947a
Compare
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jmao-dd, serathius The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
To handle the case of learner not being ready when being promoted, add retry with some delay to improve the stability of e2e tests.
This is for #19216
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.