Add audbackend.backend.Minio

[hagenw]

Implements a backend for MinIO.

MinIO is compatible with S3 storage, and we tested internally that this backend works with S3 storage on Hetzner and AWS. A test based on AWS is integrated in audeering/audb#450.

I decided to handle the authentication the same way as we do for audbackend.backend.Artifactory, by allowing to specify the username / password inside a config file or environment variable. In addition, we have the class method audbackend.backend.Minio.get_authentication() to retrieve them, in order to check their values.
For local MinIO servers we also need to set the secure argument to False. I added this to the config file as well, in oder to make it part of the backend configuration, and added the class method audbackend.backend.Minio.get_config(), which returns all entries as dictionary.
In certain edge cases you might want to add other arguments to the underlying minio.Minio class. For that reason I decided to add support for **kwargs, but do no checks if there is any conflict with authentication as I think this is only for power users anyway.

Summary by Sourcery

Add support for MinIO as a new backend in the audbackend library, enabling storage operations with MinIO. Refactor tests to accommodate the new backend and ensure robust testing of its functionality.

New Features:

• Introduce a new backend for MinIO, allowing integration with MinIO storage systems.

Enhancements:

• Refactor test parameterization to use a centralized list of backend-interface combinations, improving test maintainability.

Tests:

• Add comprehensive tests for the new MinIO backend, including authentication, file operations, and configuration parsing.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.0%. Comparing base (937da89) to head (af109fc).
Report is 1 commits behind head on main.

Additional details and impacted files

Files with missing lines	Coverage Δ
audbackend/backend/__init__.py	100.0% <100.0%> (ø)
audbackend/core/api.py	100.0% <100.0%> (ø)
audbackend/core/backend/minio.py	100.0% <100.0%> (ø)

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request implements a new backend for MinIO in the audbackend library. It adds support for MinIO storage, which is compatible with S3, and includes necessary changes to the existing codebase to integrate the new backend. The implementation follows a similar pattern to the existing Artifactory backend, with additional configuration options specific to MinIO.

Class diagram for the new Minio backend

classDiagram
    class Minio {
        - MinioClient _client
        + Minio(host: str, repository: str, authentication: Tuple[str, str] = None, secure: bool = None, **kwargs)
        + get_authentication(host: str) Tuple[str, str]
        + get_config(host: str) Dict
        + close()
        + _checksum(path: str) str
        + _collapse(path)
        + _copy_file(src_path: str, dst_path: str, verbose: bool)
        + _create()
        + _date(path: str) str
        + _delete()
        + _exists(path: str) bool
        + _get_file(src_path: str, dst_path: str, verbose: bool)
        + _ls(path: str) List[str]
        + _move_file(src_path: str, dst_path: str, verbose: bool)
        + _open()
        + _owner(path: str) str
        + path(path: str) str
        + _put_file(src_path: str, dst_path: str, checksum: str, verbose: bool)
        + _remove_file(path: str)
        + _size(path: str) int
    }
    class Base {
        <<abstract>>
    }
    Minio --|> Base
    note for Minio "This class implements a backend for MinIO storage, compatible with S3."

Loading

File-Level Changes

Change	Details	Files
Implement MinIO backend class	Create new Minio class inheriting from Base Implement authentication and configuration methods Add methods for file operations (put, get, copy, move, delete) Handle large file transfers (>5GB) with a fallback mechanism	audbackend/core/backend/minio.py
Integrate MinIO backend into existing codebase	Register MinIO backend in core API Add MinIO import in backend init.py Update test configurations to include MinIO Modify existing tests to accommodate MinIO backend	audbackend/core/api.py audbackend/backend/__init__.py tests/conftest.py tests/test_interface_unversioned.py tests/test_interface_versioned.py tests/test_interface_maven.py
Add new tests for MinIO backend	Create test_backend_minio.py with comprehensive tests Test authentication, configuration, and file operations Add tests for content type handling and large file copying	tests/test_backend_minio.py
Update existing tests to include MinIO	Modify test_api.py to include MinIO in backend tests Update test_interface_unversioned.py, test_interface_versioned.py, and test_interface_maven.py to include MinIO in interface tests	tests/test_api.py tests/test_interface_unversioned.py tests/test_interface_versioned.py tests/test_interface_maven.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey @hagenw - I've reviewed your changes and found some issues that need to be addressed.

Blocking issues:

• Hardcoded MinIO access key found. (link)
• Hardcoded MinIO secret key found. (link)

Here's what I looked at during the review

• 🟡 General issues: 1 issue found
• 🔴 Security: 2 blocking issues
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[hagenw]

I addressed now all comments and updated the description by mentioning our Hetzner S3 test.

[hagenw]

As it works with audb (see audeering/audb#450), I will merge this in order to continue with related updates.