-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Batch cancel tasks #1896
feat: Batch cancel tasks #1896
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -77,8 +77,23 @@ def get_buffer(self, file): | |
return self.buffer | ||
|
||
|
||
class BatchCancelInstanceSerializer(serializers.Serializer): | ||
id_list = serializers.ListField(required=True, child=serializers.UUIDField(required=True), | ||
error_messages=ErrMessage.char("id列表")) | ||
type = serializers.IntegerField(required=True, error_messages=ErrMessage.integer( | ||
"任务类型")) | ||
|
||
def is_valid(self, *, raise_exception=False): | ||
super().is_valid(raise_exception=True) | ||
_type = self.data.get('type') | ||
try: | ||
TaskType(_type) | ||
except Exception as e: | ||
raise AppApiException(500, '任务类型不支持') | ||
|
||
|
||
class CancelInstanceSerializer(serializers.Serializer): | ||
type = serializers.IntegerField(required=True, error_messages=ErrMessage.boolean( | ||
type = serializers.IntegerField(required=True, error_messages=ErrMessage.integer( | ||
"任务类型")) | ||
|
||
def is_valid(self, *, raise_exception=False): | ||
|
@@ -1064,6 +1079,28 @@ def batch_delete(self, instance: Dict, with_valid=True): | |
delete_embedding_by_document_list(document_id_list) | ||
return True | ||
|
||
def batch_cancel(self, instance: Dict, with_valid=True): | ||
if with_valid: | ||
self.is_valid(raise_exception=True) | ||
BatchCancelInstanceSerializer(data=instance).is_valid(raise_exception=True) | ||
document_id_list = instance.get("id_list") | ||
ListenerManagement.update_status(QuerySet(Paragraph).annotate( | ||
reversed_status=Reverse('status'), | ||
task_type_status=Substr('reversed_status', TaskType(instance.get('type')).value, | ||
1), | ||
).filter(task_type_status__in=[State.PENDING.value, State.STARTED.value]).filter( | ||
document_id__in=document_id_list).values('id'), | ||
TaskType(instance.get('type')), | ||
State.REVOKE) | ||
ListenerManagement.update_status(QuerySet(Document).annotate( | ||
reversed_status=Reverse('status'), | ||
task_type_status=Substr('reversed_status', TaskType(instance.get('type')).value, | ||
1), | ||
).filter(task_type_status__in=[State.PENDING.value, State.STARTED.value]).filter( | ||
id__in=document_id_list).values('id'), | ||
TaskType(instance.get('type')), | ||
State.REVOKE) | ||
|
||
def batch_edit_hit_handling(self, instance: Dict, with_valid=True): | ||
if with_valid: | ||
BatchSerializer(data=instance).is_valid(model=Document, raise_exception=True) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provided code has several issues and optimizations that can be addressed: Issues
Optimizations
Here's an optimized version of your code: from django.db.models import QuerySet, Substr, F, UpdateManager
from django.utils.functional import cached_property
from rest_framework import serializers, exceptions
import uuid
# Assuming these classes/variables exist somewhere in your project
class ErrMessage:
char = lambda cls, msg: {cls: f'字符格式错误: {msg}'}
integer = lambda cls, msg: {cls: f'整数格式错误: {msg}'} # Corrects typo in original
class AppApiException(exceptions.APIException):
pass
class TaskType(int):
def __new__(cls, value):
if not isinstance(value, int) or value < 0:
raise ValueError(f"无效的任务类型: {value}")
return super().__new__(cls, value)
PENDING = 0
ACTIVE = 1
INACTIVE = 2
def reverse(s):
return s[::-1]
@cached_property
def Buffer(self):
buffer = []
self.document_set.all().update(status="REVOKE")
return buffer
if __name__ == "__main__":
class CustomSerializer(serializers.Serializer):
id_list = serializers.ListField(
required=True,
child=serializers.UUIDField(required=True),
error_messages={"invalid": "ID 列表的项不是有效的 UUID"}
)
type = serializers.IntegerField(
required=True,
error_messages={"incorrect": "请求类型应为数字", "blank": "请求类型不能为空"},
validators=[lambda x: 0 <= x <= 2] # Example validator
)
def validate_type(self, value):
try:
task_type = TaskType(value)
return task_type.value
except ValueError as e:
raise serializers.ValidationError(str(e))
def save(self, instance=None, commit=True):
queryset = Document.objects.select_related('reverse')
updated_instances = queryset.annotate(
reversed_status=F('reverse').reverse(),
task_type_status=Substr(F('reversed_status'), TaskType(self.validated_data['type']).value, 2)
).filter(task_type_status='PENDI'.lower()).filter(id_in=self.validated_data['id_list']).annotate(
document_id=F('id')
).select_related('document_id')
manager = UpdateManager()
with transaction.atomic():
updates = [(inst.reverse.id, inst.task_type.status) for inst in updated_instances]
for pk, status_value in list(updates):
update_manager.filter(pk=pk).update(revoke=reverse(state=status_value))
Document.objects.bulk_update(updated_instances, fields=['status'])
serializer_instance = CustomSerializer({"id_list": ["some_uuid"], "type": 0})
(serializer_instance.is_valid(), serializer_instance.errors, serializer_instance.save()) Key Changes
This refactored code should address many of the issues identified while enhancing maintainability and security. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provided code contains several improvements, clarifications, and optimizations:
Overall, these changes improve readability, maintainability, and functionality quality. |
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -37,3 +37,17 @@ def get_request_body_api(): | |
description="1|2|3 1:向量化|2:生成问题|3:同步文档") | ||
} | ||
) | ||
|
||
class BatchCancel(ApiMixin): | ||
@staticmethod | ||
def get_request_body_api(): | ||
return openapi.Schema( | ||
type=openapi.TYPE_OBJECT, | ||
properties={ | ||
'id_list': openapi.Schema(type=openapi.TYPE_ARRAY, items=openapi.Schema(type=openapi.TYPE_STRING), | ||
title="文档id列表", | ||
description="文档id列表"), | ||
'type': openapi.Schema(type=openapi.TYPE_INTEGER, title="任务类型", | ||
description="1|2|3 1:向量化|2:生成问题|3:同步文档", default=1) | ||
} | ||
) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provided code is mostly correct and follows standard API schema definitions using OpenAPI 3 format. However, there are a couple of minor improvements that can be made:
Here's the slightly refined version of your code with these considerations addressed: class BatchCancel(ApiMixin):
@staticmethod
def get_request_body_api():
return openapi.Schema(
type=openapi.TYPE_OBJECT,
properties={
'id_list': openapi.Schema(
type=openapi.TYPE_ARRAY,
items=openapi.Schema(type=openapi.TYPE_STRING, description="文档ID列表"),
title="文档ID列表"
),
'type': openapi.Schema(
type=openapi.TYPE_INTEGER,
title="任务类型",
description="1|2|3: 向量化 | 2: 生成问题 | 3: 同步文档",
default=1
)
}
) These changes make it easier to read through and understand the structure of the request body for the |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -238,6 +238,24 @@ def put(self, request: Request, dataset_id: str, document_id: str): | |
request.data | ||
)) | ||
|
||
class Batch(APIView): | ||
authentication_classes = [TokenAuth] | ||
|
||
@action(methods=['PUT'], detail=False) | ||
@swagger_auto_schema(operation_summary="批量取消任务", | ||
operation_id="批量取消任务", | ||
request_body=DocumentApi.BatchCancel.get_request_body_api(), | ||
manual_parameters=DocumentSerializers.Create.get_request_params_api(), | ||
responses=result.get_default_response(), | ||
tags=["知识库/文档"] | ||
) | ||
@has_permissions( | ||
lambda r, k: Permission(group=Group.DATASET, operate=Operate.MANAGE, | ||
dynamic_tag=k.get('dataset_id'))) | ||
def put(self, request: Request, dataset_id: str): | ||
return result.success( | ||
DocumentSerializers.Batch(data={'dataset_id': dataset_id}).batch_cancel(request.data)) | ||
|
||
class Refresh(APIView): | ||
authentication_classes = [TokenAuth] | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provided code snippet contains several issues and areas for improvement:
To address these points, you could refactor the shared functionality by creating a base view class with common parameters and operations, then extend this base class for concrete actions like updating multiple items ( These modifications will help improve code efficiency, readability, adhere to best practices, and ensure correct behavior across your API endpoints. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provided code snippet appears to be part of an API implementation that handles CRUD (Create, Read, Update, Delete) operations on datasets and documents within a knowledge base system. There are a few minor adjustments and improvements needed:
Here's the adjusted version of the code: from rest_framework.views import APView
from django.shortcuts import render_to_response
from oauthlib.oauth2 import TokenAuth
import json
# Assuming 'result' is defined elsewhere
from .results import result
# Placeholder functions for serialization and cancel request body
class DocumentApi:
@staticmethod
def get_batch_cancel_request_body_api():
# Define the expected batch cancellation request structure here
pass
# Placeholder function for document serializations
class DocumentSerializers:
@staticmethod
def Create(get_request_params_api):
# Define the create serializer here
pass
@staticmethod
def BatchCancel(batch_cancellation_serializer):
# Define the batch cancellation serializer here
pass
def put(self, request: Request, dataset_id: str, document_id: str):
return result.success(DocumentSerializers.Create().create_document(request.data))
class Batch(APIView):
authentication_classes = [TokenAuth]
@action(methods=['PUT'], detail=False)
@swagger_auto_schema(
operation_summary="批量取消任务",
operation_id="批量取消任务",
request_body=DocumentApi.BatchCancel.get_batch_cancel_request_body_api(),
manual_parameters=[
ManualParameter(name='dataset_id', location='path', type=str),
*DocumentSerializers.Create.get_request_params_api()
],
responses=result.get_default_response(),
tags=["知识库/文档"]
)
@has_permissions(
lambda r, k: Permission(group=Group.DATASET, operate=Operate.MANAGE,
dynamic_tag=k.get('dataset_id')))
def put(self, request: Request, dataset_id: str):
return result.success(
DocumentSerializers.BatchCancel().batch_cancel(request.data))
class Refresh(APIView):
authentication_classes = [TokenAuth] Key Changes:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There appears to be a few areas that need attention:
Here's a corrected version of your code: @@ -238,6 +238,25 @@ def put(self, request: Request, dataset_id: str, document_id: str):
request.data
))
class Batch(APIView):
authentication_classes = [TokenAuth]
- @action(methods=['PUT'], detail=False)
+ @action(methods=['PUT'], detail=False)
+ @swagger_auto_schema(operation_summary="批量取消任务",
+ operation_id="批量取消任务",
+ request_body=DocumentApi.BatchCancel.get_request_body_api(),
+ manual_parameters=DocumentSerializers.Create.get_request_params_api(),
+ responses=result.get_default_response(),
+ tags=["知识库/文档"]
+ )
+ @has_permissions(
+ lambda r, k: Permission(group=Group.DATASET, operate=Operate.MANAGE,
+ dynamic_tag=k.get('dataset_id')))
+ def put(self, request: Request, dataset_id: str) -> JsonResponse:
+ # Ensure you are returning a JsonResponse object here
+ return result.success(
+ DocumentSerializers.Batch(data={'dataset_id': dataset_id}).batch_cancel(request.data))
class Refresh(APIView):
authentication_classes = [TokenAuth] Key Corrections:
These changes should resolve any syntax errors or misconfigurations in your code. |
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
<template> | ||
<LayoutContainer header="文档"> | ||
<LayoutContainer header="文档" class="document-main"> | ||
<div class="main-calc-height"> | ||
<div class="p-24"> | ||
<div class="flex-between"> | ||
|
@@ -403,12 +403,25 @@ | |
</el-table-column> | ||
</app-table> | ||
</div> | ||
|
||
<ImportDocumentDialog ref="ImportDocumentDialogRef" :title="title" @refresh="refresh" /> | ||
<SyncWebDialog ref="SyncWebDialogRef" @refresh="refresh" /> | ||
<!-- 选择知识库 --> | ||
<SelectDatasetDialog ref="SelectDatasetDialogRef" @refresh="refreshMigrate" /> | ||
<GenerateRelatedDialog ref="GenerateRelatedDialogRef" @refresh="refresh" /> | ||
</div> | ||
<div class="mul-operation w-full flex" v-if="multipleSelection.length !== 0"> | ||
<el-button :disabled="multipleSelection.length === 0" @click="cancelTaskHandle(1)"> | ||
取消向量化 | ||
</el-button> | ||
<el-button :disabled="multipleSelection.length === 0" @click="cancelTaskHandle(2)"> | ||
取消生成 | ||
</el-button> | ||
<el-text type="info" class="secondary ml-24"> | ||
已选 {{ multipleSelection.length }} 项 | ||
</el-text> | ||
<el-button class="ml-16" type="primary" link @click="clearSelection"> 清空 </el-button> | ||
</div> | ||
</LayoutContainer> | ||
</template> | ||
<script setup lang="ts"> | ||
|
@@ -478,6 +491,7 @@ const multipleSelection = ref<any[]>([]) | |
const title = ref('') | ||
|
||
const SelectDatasetDialogRef = ref() | ||
|
||
const exportDocument = (document: any) => { | ||
documentApi.exportDocument(document.name, document.dataset_id, document.id, loading).then(() => { | ||
MsgSuccess('导出成功') | ||
|
@@ -490,6 +504,28 @@ const exportDocumentZip = (document: any) => { | |
MsgSuccess('导出成功') | ||
}) | ||
} | ||
|
||
function cancelTaskHandle(val: any) { | ||
const arr: string[] = [] | ||
multipleSelection.value.map((v) => { | ||
if (v) { | ||
arr.push(v.id) | ||
} | ||
}) | ||
const obj = { | ||
id_list: arr, | ||
type: val | ||
} | ||
documentApi.batchCancelTask(id, obj, loading).then(() => { | ||
MsgSuccess('批量取消成功') | ||
multipleTableRef.value?.clearSelection() | ||
}) | ||
} | ||
|
||
function clearSelection() { | ||
multipleTableRef.value?.clearSelection() | ||
} | ||
|
||
function openDatasetDialog(row?: any) { | ||
const arr: string[] = [] | ||
if (row) { | ||
|
@@ -813,4 +849,20 @@ onBeforeUnmount(() => { | |
closeInterval() | ||
}) | ||
</script> | ||
<style lang="scss" scoped></style> | ||
<style lang="scss" scoped> | ||
.document-main { | ||
box-sizing: border-box; | ||
.mul-operation { | ||
position: fixed; | ||
margin-left: var(--sidebar-width); | ||
bottom: 0; | ||
right: 24px; | ||
width: calc(100% - var(--sidebar-width) - 48px); | ||
padding: 16px 24px; | ||
box-sizing: border-box; | ||
background: #ffffff; | ||
z-index: 22; | ||
box-shadow: 0px -2px 4px 0px rgba(31, 35, 41, 0.08); | ||
} | ||
} | ||
</style> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The provided code appears to be primarily template and script components of an SFC (Single File Component) with Vue.js structure. Here are some minor points of analysis regarding its correctness, potential issues, and optimizations: Points:
Optimization Advice:Move the <style lang="scss">
/* ... existing inline SCSS here ... */
</style>
Potential Issues:While not directly related to this snippet, ensuring proper naming conventions and avoiding overuse of generic classes can help maintain clarity and prevent confusion when dealing with larger projects.
Code Review:There seems to be no apparent issues with the logic implemented using refs ( Additional Recommendations:
After making appropriate changes according to the recommendations above, test thoroughly to ensure behavior meets expectations and adheres to best practices in Vue.js development. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The code seems syntactically correct and follows good practices. However, a few minor improvements can be made for readability and maintainability:
Here’s the slightly improved version of the code with these considerations: <script setup lang="ts">
import { ref } from 'vue'
import LayoutContainer from '@/components/LayoutContainer.vue' // Adjust path based on actual implementation
const multipleSelection = ref<any[]>([])
const title = ref('')
const SelectDatasetDialogRef = ref()
const exportDocument = async (document: any) => {
await documentApi.exportDocument(document.name, document.dataset_id, document.id, loading).then(() => {
MsgSuccess('导出成功')
})
}
const exportDocumentZip = async (document: any) => {
await documentApi.exportDocumentZip(document.name, document.dataset_id, document.id, loading).then(() => {
MsgSuccess('导出成功')
})
}
function cancelTaskHandle(val: number) {
const arr: string[] = []
multipleSelection.value.forEach((v) => {
if (v && typeof v.id !== 'undefined') {
arr.push(v.id)
}
})
const obj = {
id_list: arr,
type: val
}
documentApi.batchCancelTask(id, obj, loading).then(() => {
MsgSuccess('批量取消成功')
multipleTableRef.value?.clearSelection()
})
}
function clearSelection() {
multipleTableRef.value?.clearSelection()
}
// Function definitions remain unchanged...
onBeforeUnmount(() => {
closeInterval()
})
</script>
<style scoped lang="scss">
.document-main {
box-sizing: border-box;
.mul-operation {
position: fixed;
left: var(--sidebar-width); /* Adjusted positioning logic */
bottom: 0;
right: 24px;
width: calc(100% - var(--sidebar-width) - 48px);
padding: 16px 24px;
box-sizing: border-box;
background: #ffffff;
z-index: 22;
box-shadow: 0px -2px 4px 0px rgba(31, 35, 41, 0.08);
}
}
</style> These changes enhance readability by improving naming consistency and clarifying some logical steps within the code. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The provided code has several issues that need to be addressed:
BatchCancelInstanceSerializer: The validation in this serializer does not properly handle cases where the
id_list
contains non-UUID values or if elements ofid_list
do not match the expected format.Additionally, ensure that
ErrorMessages.char
,ErrorMessages.integer
, and other error messages used throughout the code are defined correctly and consistently within your project context.batch_cancel method: Improper handling of tasks whose status might already be REVOKE or PENDING/STARTED before cancelling them could lead to redundant updates.
Consider adding more comprehensive checks to prevent unwanted operations like canceling an already canceled or ongoing transaction. For example:
Ensure this additional function logic complies with your application's specific requirements for permission controls and logging.
Additionally, review how exception propagation works during API calls – make sure it doesn't result in unexpected behavior when there’s a data integrity issue affecting multiple instances simultaneously. This might require careful attention to rollback transactions across different models or using proper error management strategies.