Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DDL job state change might be overridden #52747

Closed
D3Hunter opened this issue Apr 19, 2024 · 0 comments · Fixed by #58495 · May be fixed by #53518
Closed

DDL job state change might be overridden #52747

D3Hunter opened this issue Apr 19, 2024 · 0 comments · Fixed by #58495 · May be fixed by #53518
Assignees
@D3Hunter
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

D3Hunter commented Apr 19, 2024 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

below is the step when cancel operation is overridden. in real, any state change might be overridden on owner change cases where 2 owner might tries to change at same time, if the TXNs don't cross with each other, the prior one is overridden.

  • apply below batch
  • cancel the job before we start running it
diff --git a/pkg/ddl/ddl.go b/pkg/ddl/ddl.go
index 51fb9a9434..129f9cc00a 100644
--- a/pkg/ddl/ddl.go
+++ b/pkg/ddl/ddl.go
@@ -1724,6 +1724,8 @@ func resumePausedJob(_ *sess.Session, job *model.Job,
 	return nil
 }
 
+var GLOBAL = make(chan struct{})
+
 // processJobs command on the Job according to the process
 func processJobs(process func(*sess.Session, *model.Job, model.AdminCommandOperator) (err error),
 	sessCtx sessionctx.Context,
@@ -1794,6 +1796,8 @@ func processJobs(process func(*sess.Session, *model.Job, model.AdminCommandOpera
 		if err = ns.Commit(); err != nil {
 			continue
 		}
+		fmt.Println("after cancel job-------------------")
+		GLOBAL <- struct{}{}
 
 		for id, idx := range jobMap {
 			jobErrs[idx] = dbterror.ErrDDLJobNotFound.GenWithStackByArgs(id)
diff --git a/pkg/ddl/ddl_worker.go b/pkg/ddl/ddl_worker.go
index 6f6084fa61..dc84e45f05 100644
--- a/pkg/ddl/ddl_worker.go
+++ b/pkg/ddl/ddl_worker.go
@@ -853,6 +853,12 @@ func (w *worker) HandleJobDone(d *ddlCtx, job *model.Job, t *meta.Meta) error {
 }
 
 func (w *worker) prepareTxn(job *model.Job) (kv.Transaction, error) {
+	if job.Type == model.ActionTruncateTable && job.State == model.JobStateQueueing {
+		fmt.Println("before truncate -------------------", job.ID)
+		<-GLOBAL
+		time.Sleep(10 * time.Second)
+		fmt.Println("start truncate -------------------", job.ID)
+	}
 	err := w.sess.Begin()
 	if err != nil {
 		return nil, err

2. What did you expect to see? (Required)

job cancelled

3. What did you see instead (Required)

job cancel success, but the job keeps running as the meta is overridden

mysql> admin cancel ddl jobs 136;
+--------+------------+
| JOB_ID | RESULT     |
+--------+------------+
| 136    | successful |
+--------+------------+
1 row in set (0.00 sec)

mysql> admin show ddl jobs 1;
+--------+---------+------------+----------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+------------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE       | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE      |
+--------+---------+------------+----------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+------------+
|    136 | test    | t          | truncate table | none         |         2 |      133 |         0 | 2024-04-18 23:30:30 | NULL                | NULL                | cancelling |
|    134 | test    | t          | truncate table | public       |         2 |      131 |         0 | 2024-04-18 23:26:27 | 2024-04-18 23:26:54 | 2024-04-18 23:30:40 | synced     |
+--------+---------+------------+----------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+------------+
2 rows in set (0.00 sec)

mysql> admin show ddl jobs 1;
+--------+---------+------------+----------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
| JOB_ID | DB_NAME | TABLE_NAME | JOB_TYPE       | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE  |
+--------+---------+------------+----------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
|    136 | test    | t          | truncate table | public       |         2 |      133 |         0 | 2024-04-18 23:30:30 | 2024-04-18 23:31:34 | 2024-04-18 23:31:34 | synced |
+--------+---------+------------+----------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
1 row in set (0.01 sec)

4. What is your TiDB version? (Required)

master
@D3Hunter D3Hunter added the type/bug The issue is confirmed as a bug. label Apr 19, 2024
@jebter jebter added component/ddl This issue is related to DDL of TiDB. severity/major labels Apr 19, 2024
@tangenta tangenta self-assigned this May 23, 2024
@tangenta tangenta mentioned this issue May 24, 2024
Open
13 tasks
@D3Hunter D3Hunter mentioned this issue Oct 12, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot added the affects-8.5 This bug affects the 8.5.x(LTS) versions. label Nov 1, 2024
@D3Hunter D3Hunter mentioned this issue Dec 5, 2024
Open
@D3Hunter D3Hunter mentioned this issue Dec 23, 2024
Open
@D3Hunter D3Hunter changed the title DDL job cancel operation might be overridden DDL job state change might be overridden Dec 23, 2024
@D3Hunter D3Hunter assigned D3Hunter and unassigned tangenta Dec 23, 2024
@D3Hunter D3Hunter added affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-6.5 This bug affects the 6.5.x(LTS) versions. and removed may-affects-6.1 may-affects-6.5 may-affects-7.1 may-affects-7.5 may-affects-8.1 may-affects-5.4 This bug maybe affects 5.4.x versions. labels Dec 24, 2024
@D3Hunter D3Hunter mentioned this issue Dec 24, 2024
Merged
13 tasks
@ti-chi-bot ti-chi-bot bot closed this as completed in 46aa33b Dec 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@D3Hunter D3Hunter

Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. affects-7.5 This bug affects the 7.5.x(LTS) versions. affects-8.1 This bug affects the 8.1.x(LTS) versions. affects-8.5 This bug affects the 8.5.x(LTS) versions. component/ddl This issue is related to DDL of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
3 participants
@D3Hunter @jebter @tangenta