-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce cron jobs to clean up the export cache (data/cache/export/
) and temporary files/directories (data/tmp/
)
#8804
Conversation
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThe pull request introduces a comprehensive refactoring of the export and cache management system in the CVAT application. The changes span multiple files and focus on improving the modularity, error handling, and organization of export-related functionality. Key modifications include introducing new classes for file type management, restructuring export cache handling, adding periodic cleanup jobs for export caches, and enhancing the overall file management process across projects, tasks, and jobs. Changes
Poem
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
b885dac
to
f20eadf
Compare
f20eadf
to
92f9d0b
Compare
/check |
❌ Some checks failed |
14084b8
to
e4c24ea
Compare
/check |
✔️ All checks completed successfully |
@@ -542,7 +559,7 @@ def _export_task(self, zip_obj, target_dir=None): | |||
self._write_annotations(zip_obj, target_dir) | |||
self._write_annotation_guide(zip_obj, target_dir) | |||
|
|||
def export_to(self, file, target_dir=None): | |||
def export_to(self, file: str | ZipFile, target_dir: str | None = None): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not an error, but such annotations in inherited functions tend to become outdated quickly. Typically it's better to just use annotations from the original function instead of duplicating them everywhere.
cvat/apps/dataset_manager/management/commands/exportcachecleanup.py
Outdated
Show resolved
Hide resolved
cvat/apps/dataset_manager/management/commands/exportcachecleanup.py
Outdated
Show resolved
Hide resolved
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
cvat/apps/dataset_manager/management/commands/exportcachecleanup.py
Outdated
Show resolved
Hide resolved
e5b1236
to
6baf90b
Compare
Quality Gate passedIssues Measures |
Motivation and context
Depends on #8721PR introduces the following changes:
data/tmp/
) instead ofproject|task|job/id/tmp/export_cache/
exportcachecleanup
management command to remove outdatedproject|task|job/id/tmp/export_cache/
directoriesdata/tmp/
directory. The new settingTMP_FILE_OR_DIR_RETENTION_DAYS
is used to determine whether a file or directory should be removedNotes:
data/project|task|job/id/tmp/
is still used during uploading annotations/datasets, but this should be fixed in a separate PR.Breaking changes:
cvat.apps.dataset_manager.views.clear_export_cache
was moved andcvat.apps.engine.backup._clear_export_cache
was deleted.How has this been tested?
Checklist
develop
branch(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.
Summary by CodeRabbit
New Features
Bug Fixes
Refactor
Documentation
Chores