Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with AWS IRSA Configuration for Restore: S3 Object Header Forbidden (403) #1616

Open
HamoucheTF1 opened this issue Aug 6, 2024 · 1 comment
Labels

Comments

@HamoucheTF1
Copy link

HamoucheTF1 commented Aug 6, 2024

Report

When configuring AWS IAM Roles for Service Accounts (IRSA) for backups in the Percona MongoDB Operator, the backup process works correctly. However, restoring from the backup fails with a "Forbidden" error when trying to access S3 objects.

More about the problem

"error": "set resync backup list from the store: init storage: get S3 object header: Forbidden: Forbidden\n\tstatus code: 403"

Steps to reproduce

  1. Configure AWS IRSA:
    Set up IAM roles and policies for the Percona MongoDB Operator.
    Annotate the Kubernetes service account with the IAM role ARN.
    refer to this doumentation: https://docs.percona.com/percona-backup-mongodb/manage/automate-s3-access.html#iam-roles-for-service-accounts-irsa
  2. Configure Backup:
    Set up the Percona MongoDB helm chart to use IRSA for S3 backups.
    backups are created and listed successfully.
    backups are configured with psmdb-db helm chart
backup:
  enabled: true
  pitr:
    enabled: false
  storages:
    s3-us-west:
      type: s3
      s3:
        bucket: S3-BACKUP-BUCKET-NAME-HERE
        credentialsSecret: "" # empty and it works, the backup use the aws IRSA
        prefix: "someprefix"
        region: us-west-3
  1. Initiate a restore process using the PerconaServerMongoDBRestore CRD.
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore1
spec:
  clusterName: my-cluster-name
  backupName: backup1
  storageName: s3-us-west
 backupSource:
   type: logical
   storageName: s3-us-west
   destination: s3://S3-BACKUP-BUCKET-NAME-HERE/BACKUP-DESTINATION
   s3:
     credentialsSecret: "" # empty, i want to use the aws IRSA
     region: us-west-2
     bucket: S3-BACKUP-BUCKET-NAME-HERE
     prefix: "some prefix"

Versions

  1. Kubernetes: 1.29
  2. Operator: 1.16.1
  3. Helm chart psmdb-db: 1.16.2
  4. Database: mongodb 7.0.8-5

Anything else?

No response

@HamoucheTF1 HamoucheTF1 added the bug label Aug 6, 2024
@hors
Copy link
Collaborator

hors commented Nov 15, 2024

Hi @HamoucheTF1, PSMDBO 1.18.0 with PBM 2.7.0 supports IRSA, but the operator's service account also should have access to the bucket because the operator performs backup validation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants