Update Operator & Tenant at the same time fails

[tmohme]

I try to update mini-operator from 6.0.3 to 7.0.0 and mini-tenant from RELEASE.2024-08-26T15-33-07Z to the current RELEASE.2025-01-20T14-49-07Z.
The environment is K8s with ArgoCD deploying Helm-Charts.

This is more or less the same as #2305.

Expected Behavior

I expect the mini-operator to update to 7.0.0 and the operator to pick-up the change in the Tenant CR from "spec.image=quay.io/minio/minio:RELEASE.2024-08-26T15-33-07Z" to "spec.image=quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z", subsequently change the Tenant-Pool-StatefulSet and finally restarting its pods so that they run with the new image.

Current Behavior

The mini-operator successfully updates to 7.0.0 and the Tenant CR is also changed to "spec.image=quay.io/minio/minio:RELEASE.2025-01-20T14-49-07Z" as expected.
But then the operate seems to send an "update" to the minio servers, which fail to execute it due to missing access rights/wrong image layout:
server update failed with: open /usr/bin/.minio.check-perm: permission denied, do not restart the servers yet.

Possible Solution

A workaround is, to simply kill the STS and let the operator re-create it, then with the correct minio-image.
A solution would be, to let the operator patch the STS and restart the pods.

When I keep the operator version 6.0.3 and only update the minio image, the operator continuously logs an error message
main-controller.go:1570] error synching 'minio/minio-tenant': host mini-tenant-pool01-1.minio-tenant-hl.minio.svc.cluster.local:9000: Server is running the latest version: 2025-01-20T14:49:07Z;... and the status.currentStatus of the Tenant CR alternating between Updating MinIO Version and host mini-tenant-pool01-1.minio-tenant-hl.minio.svc.cluster.local:9000: Server is running the latest version: 2025-01-20T14:49:07Z;..., never getting "ready".
Therefore, this is not a solution for us.

Steps to Reproduce (for bugs)

1. Deploy minio-operator 6.0.3
2. Deploy minio:RELEASE.2024-08-26T15-33-07Z
3. Update the operate Helm chart to 7.0.0 and minio to RELEASE.2025-01-20T14-49-07Z

Context

Consequently using GitOps and infrastructure-as-code we don't want to modify running images at all.
Instead we want to configure a new version of the app and replace the old pod with the new ones to ensure full reproducibility.

Regression

don't know

Your Environment

• Version used (minio-operator): 6.0.3 => 7.0.0
• Environment name and version (e.g. kubernetes v1.17.2): v1.31.0
• Server type and version:
• Operating System and version (uname -a): Linux mini-tenant-pool01-1 6.12.9-talos Add Minio operator  #1 SMP x86_64 GNU/Linux
• Link to your deployment file: