v1.15 Velero with v1.8 AWS plugin cannot support multiple BSLs under AWS IRSA environment.

[blackpiglet]

What steps did you take and what happened:

• Create an EKS cluster, and deploy a MinIO as the backend of the additional BSL.
• Install Velero with the AWS S3 as the backend of the default BSL and use the IRSA as the authorization method. The Velero version is the main(v1.15) and the Velero AWS plugin version is v1.8.2
• Create an additional BSL with the MinIO as the backend, and create a secret as the authorization for the additional BSL.
• The additional BSL cannot work. The error is

"BackupStorageLocation \"add-bsl\" is unavailable: rpc error: code = Unknown desc = WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post \"https://sts.minio.amazonaws.com/\": dial tcp: lookup sts.minio.amazonaws.com on 10.100.0.10:53: no such host",

What did you expect to happen:
The additional BSL should work.

The following information will help us better understand what's going on:
The reason is that Velero overrides the secret for the additional BSL when IRSA is enabled.
The v1.9 and main Velero AWS plugin have this PR to fix the issue: vmware-tanzu/velero-plugin-for-aws#191
But the v1.8 doesn't bump the AWS SDK version to v2, so we cannot simply cherry-pick the PR to fix this issue.

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[blackpiglet]

@kaovilai @shubham-pampattiwar @reasonerjt

[kaovilai]

Is it common to have multiple minio BSLs with IRSA?

[blackpiglet]

The IRSA scenario also has an additional secret-based BSL, which may not be a common case.

[kaovilai]

Sounds like easy enough port.. unsetting envvar. If you want me to try PR (although never used IRSA) lmk.

[reasonerjt]

IMO probably this can be a low priority b/c we verify the AWS plugin v1.8 for S3-compatible storage, i.e. non-AWS storage, but IRSA only exists in AWS.

@kaovilai what do you think?

[kaovilai]

yeah if 1.8 plugin is only for s3-compatible and we intend to maintain v1.10 plugin, we can ignore IRSA for aws via 1.8 plugin.

[blackpiglet]

I closed this issue for now because this corner case is acceptable.