feat: add delete task and list tasks manager api with request type and service type. #3378

IRONICBo · 2024-07-15T11:54:13Z

Description

Add a POST method to start an asynchronous task to clear the cache of the task id on the peer, and a query api to query the status of the asynchronous task and aggregate it into the existing asynchronous task creation logic.
The manager eventually collects the information needed to start the job and creates a GroupJob into the machine to send a message to the distributed task waiting for the scheduler to consume it.
There is already a Job definition that can reuse the job request to support a new Job, define a new JobRequest, and the response reuses the Job response.

Meanwhile, this PR add a delete task job and list tasks job implementation to the scheduler's async job to receive incoming jobs from the manager and call grpc's interface to complete tasks queries and deletions.

Related Issue

Manager console provides clearing the cache for P2P with task id #2818

Motivation and Context

This change introduces api definitions for delete task and list task and request responsetype definitions, as well as interfaces under the services package, and introduced internaljob configuration parameters and job calls to asynchronously list or delete peers and task information for associated tasks in the scheduler.

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

Checklist

My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.

codecov · 2024-07-23T02:50:32Z

Codecov Report

Attention: Patch coverage is 12.50000% with 77 lines in your changes missing coverage. Please review.

Project coverage is 52.54%. Comparing base (83450d5) to head (915d880).
Report is 14 commits behind head on main.

Files	Patch %	Lines
manager/job/manager_tasks.go	0.00%	50 Missing ⚠️
manager/handlers/job.go	40.00%	8 Missing and 4 partials ⚠️
scheduler/resource/task.go	0.00%	10 Missing ⚠️
manager/job/job.go	0.00%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3378      +/-   ##
==========================================
- Coverage   52.76%   52.54%   -0.22%     
==========================================
  Files         193      194       +1     
  Lines       20485    20578      +93     
==========================================
+ Hits        10808    10813       +5     
- Misses       8873     8955      +82     
- Partials      804      810       +6

Flag	Coverage Δ
unittests	`52.54% <12.50%> (-0.22%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
manager/config/config.go	`96.20% <100.00%> (+0.05%)`	⬆️
manager/job/job.go	`0.00% <0.00%> (ø)`
scheduler/resource/task.go	`78.47% <0.00%> (-3.69%)`	⬇️
manager/handlers/job.go	`63.85% <40.00%> (-7.58%)`	⬇️
manager/job/manager_tasks.go	`0.00% <0.00%> (ø)`

... and 6 files with indirect coverage changes

Signed-off-by: Asklv <[email protected]>

…st tasks. Signed-off-by: Asklv <[email protected]>

Signed-off-by: Asklv <[email protected]>

chlins · 2024-08-02T03:27:35Z

manager/job/manager_tasks.go

+
+	logger.Infof("create manager tasks group %s in queues %v, tasks: %#v", group.GroupUUID, queues, tasks)
+	if _, err := m.job.Server.SendGroupWithContext(ctx, group, 0); err != nil {
+		logger.Errorf("create  manager tasks group %s failed", group.GroupUUID, err)


Remove extra spaces in log.

chlins · 2024-08-02T03:34:59Z

manager/types/job.go

+
+type ListTasksArgs struct {
+	TaskID  string `json:"task_id" binding:"required"`
+	Page    int    `json:"page" binding:"omitempty,gte=1"`


Page and PerPage are required by many requests. We can define a common request argument to reuse, but that's not the focus of this PR. We can address it in a subsequent refactoring.

Well, there is a page and perpage parameter in the task itself, but this is mainly for handling page and perpage in the structure returned within the task result, and I'll consider defining a generic version in subsequent implementations as well.

…ntation. Signed-off-by: Asklv <[email protected]>

Signed-off-by: Asklv <[email protected]>

chlins · 2024-08-05T12:12:36Z

scheduler/job/job.go

+
+// listTasks is a job to list tasks.
+func (j *job) listTasks(ctx context.Context, data string) (string, error) {
+	ctx, cancel := context.WithTimeout(ctx, listTasksTimeout)


I didn't see any reference or usage for the ctx in this method.

Ok, this file should belong to another scheduler PR, which I have currently removed.

And I have fixed this ctx usage in another PR.

chlins · 2024-08-05T12:13:18Z

scheduler/job/job.go

+
+// deleteTask is a job to delete task.
+func (j *job) deleteTask(ctx context.Context, data string) (string, error) {
+	ctx, cancel := context.WithTimeout(ctx, deleteTaskTimeout)


chlins

lgtm

…ntation. Signed-off-by: Asklv <[email protected]>

…ic implement. Signed-off-by: Asklv <[email protected]>

…er valid tasks. Signed-off-by: Asklv <[email protected]>

gaius-qi · 2024-08-09T04:23:57Z

scheduler/job/job.go

+}
+
+// getValidPeers try to get valid peers by task id
+func (j *job) getValidPeers(taskID string) ([]*resource.Peer, error) {


getFinishedPeers

OK, I have updated this function's name.

gaius-qi · 2024-08-09T04:30:27Z

scheduler/job/job.go

+	}
+
+	// get peer info by task info
+	peers := make([]*resource.Peer, 0)


Add LoadFinishedPeers to task.

OK, moved to scheduler/resource/task.go::LoadFinishedPeers.

gaius-qi · 2024-08-09T04:35:22Z

scheduler/job/job.go

+
+	// Create a wait group to limit delete rpc concurrency
+	// and avoid too many rpc requests to the host.
+	wg := sync.WaitGroup{}


No concurrent deletion.

OK, removed.

gaius-qi · 2024-08-09T04:36:48Z

scheduler/job/job.go

+
+			// Get dfdaemon client from host
+			target := fmt.Sprintf("%s:%d", peer.Host.IP, peer.Host.Port)
+			conn, err := grpc.DialContext(


Use pkg/rpc client.

OK, I have added a GetV2ByAddr() function in pkg/rpc/dfdaemon/client/client_v2.go, refer to pkg/rpc/manager/client/client_v2.go::GetV2ByAddr()

Signed-off-by: Asklv <[email protected]>

gaius-qi

LGTM

gaius-qi · 2024-08-12T02:49:52Z

@IRONICBo Fix Lint.

gaius-qi

LGTM

IRONICBo requested a review from a team as a code owner July 15, 2024 11:54

IRONICBo requested review from imeoer, anjia0532 and cndoit18 July 15, 2024 11:54

IRONICBo force-pushed the feat/add-delete-task-manager-api branch 2 times, most recently from 105e622 to a576067 Compare July 15, 2024 11:59

IRONICBo force-pushed the feat/add-delete-task-manager-api branch from 38d3b15 to 987e070 Compare July 24, 2024 01:54

IRONICBo added 3 commits July 30, 2024 10:44

feat: add delete task and list tasks manager api define.

64420a9

Signed-off-by: Asklv <[email protected]>

feat: add manager job service and service mock for delete task and li…

b3d4e17

…st tasks. Signed-off-by: Asklv <[email protected]>

chore: fix lint issue and generate the mocked job file.

df2ae9c

Signed-off-by: Asklv <[email protected]>

IRONICBo force-pushed the feat/add-delete-task-manager-api branch from 987e070 to df2ae9c Compare July 30, 2024 02:45

chlins reviewed Aug 2, 2024

View reviewed changes

IRONICBo added 2 commits August 4, 2024 10:08

feat: add delete task job and list tasks job in scheduler job impleme…

dcbb0f7

…ntation. Signed-off-by: Asklv <[email protected]>

fix: update delete task job and list tasks job api args.

16cb224

Signed-off-by: Asklv <[email protected]>

IRONICBo force-pushed the feat/add-delete-task-manager-api branch 2 times, most recently from 77ac45c to 7aee5fd Compare August 5, 2024 11:12

IRONICBo requested a review from chlins August 5, 2024 11:50

chlins reviewed Aug 5, 2024

View reviewed changes

IRONICBo force-pushed the feat/add-delete-task-manager-api branch from f4dd25c to dcbb0f7 Compare August 5, 2024 12:22

IRONICBo requested a review from chlins August 5, 2024 12:24

chlins previously approved these changes Aug 9, 2024

View reviewed changes

IRONICBo dismissed chlins’s stale review via 16cb224 August 9, 2024 02:49

IRONICBo force-pushed the feat/add-delete-task-manager-api branch from 572d87c to 16cb224 Compare August 9, 2024 02:49

IRONICBo added 3 commits August 9, 2024 10:58

feat: add delete task job and list tasks job in scheduler job impleme…

16e73d6

…ntation. Signed-off-by: Asklv <[email protected]>

feat: add delete task job and list tasks job scheduler define and bas…

a212060

…ic implement. Signed-off-by: Asklv <[email protected]>

feat: update delete task job and list tasks with concurrency and filt…

eae443a

…er valid tasks. Signed-off-by: Asklv <[email protected]>

IRONICBo requested a review from chlins August 9, 2024 03:11

IRONICBo mentioned this pull request Aug 9, 2024

feat: add delete task job and list tasks job in scheduler job implementation. #3401

Closed

8 tasks

gaius-qi requested changes Aug 9, 2024

View reviewed changes

IRONICBo requested a review from gaius-qi August 10, 2024 13:22

fix: update getFinishedPeers function and remove concurrency feature.

915d880

Signed-off-by: Asklv <[email protected]>

IRONICBo force-pushed the feat/add-delete-task-manager-api branch from 323c9fb to 2b2d3fd Compare August 10, 2024 13:24

gaius-qi previously approved these changes Aug 12, 2024

View reviewed changes

IRONICBo dismissed gaius-qi’s stale review via 915d880 August 12, 2024 03:40

IRONICBo force-pushed the feat/add-delete-task-manager-api branch from 2b2d3fd to 915d880 Compare August 12, 2024 03:40

gaius-qi approved these changes Aug 13, 2024

View reviewed changes

gaius-qi merged commit 5547307 into dragonflyoss:main Aug 13, 2024
29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add delete task and list tasks manager api with request type and service type. #3378

feat: add delete task and list tasks manager api with request type and service type. #3378

IRONICBo commented Jul 15, 2024 •

edited

Loading

codecov bot commented Jul 23, 2024 •

edited

Loading

chlins Aug 2, 2024

IRONICBo Aug 5, 2024

chlins Aug 2, 2024

IRONICBo Aug 5, 2024

chlins Aug 5, 2024

IRONICBo Aug 5, 2024 •

edited

Loading

chlins Aug 5, 2024

IRONICBo Aug 5, 2024

chlins left a comment

gaius-qi Aug 9, 2024

IRONICBo Aug 10, 2024

gaius-qi Aug 9, 2024

IRONICBo Aug 10, 2024

gaius-qi Aug 9, 2024

IRONICBo Aug 10, 2024

gaius-qi Aug 9, 2024

IRONICBo Aug 10, 2024 •

edited

Loading

gaius-qi left a comment

gaius-qi commented Aug 12, 2024

gaius-qi left a comment

feat: add delete task and list tasks manager api with request type and service type. #3378

feat: add delete task and list tasks manager api with request type and service type. #3378

Conversation

IRONICBo commented Jul 15, 2024 • edited Loading

Description

Related Issue

Motivation and Context

Screenshots (if appropriate)

Types of changes

Checklist

codecov bot commented Jul 23, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IRONICBo Aug 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chlins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

IRONICBo Aug 10, 2024 • edited Loading

Choose a reason for hiding this comment

gaius-qi left a comment

Choose a reason for hiding this comment

gaius-qi commented Aug 12, 2024

gaius-qi left a comment

Choose a reason for hiding this comment

IRONICBo commented Jul 15, 2024 •

edited

Loading

codecov bot commented Jul 23, 2024 •

edited

Loading

IRONICBo Aug 5, 2024 •

edited

Loading

IRONICBo Aug 10, 2024 •

edited

Loading