[efs-controller] Encrypted EFS filesystem never reaches synced status

[gnadaban]

Describe the bug
I'm trying to create a FileSystem resource that is encrypted with a custom KMS key with accompanying MountTargets and AccessPoint, but there are various issues:

• The FileSystem CR is stuck in creating state while in AWS console the resource appears ready and available
• If using the fileSystemRef for MountTarget and AccessPoint resources they are never created as the controller does not consider the target FileSystem resource ready ("the referenced resource is not synced yet.")

Steps to reproduce

1. Attempt to create a FileSystem resource
2. Attempt to create MountTarget and AccessPoint resources with fileSystemRef pointing at FileSystem resource
3. FileSystem resource state is stuck in "creating"

Expected outcome

1. The FileSystem resource should become "ready" after it is successfully created and its status in AWS console is "Available".
2. The MountTarget and AccessPoint resources should be created as soon as the FileSystem resource is created and has an ARN

Environment

• Kubernetes version: v1.30.4-eks-a737599
• Using EKS (yes/no), if so version? yes
• AWS service targeted (S3, RDS, etc.) EFS

[Yunbo-Lu]

I encountered the similar issue.

When I deploy the FileSytem manfiest like below, the STATE status is stuck at creating and SYNCED status is stuck at False:

apiVersion: efs.services.k8s.aws/v1alpha1
kind: FileSystem
metadata:
  name: my-filesystem
  namespace: ack-system
spec:
  encrypted: true
  throughputMode: elastic
  performanceMode: generalPurpose
  lifecyclePolicies:
    - transitionToIA: AFTER_30_DAYS
  tags:
    - key: Name
      value: MyEFSFileSystem
    - key: Environment
      value: Production

When I deploy the MountTarget manfiest like below, the SYNCED status is True but the STATE status is stuck at creating:

---
apiVersion: efs.services.k8s.aws/v1alpha1
kind: MountTarget
metadata:
  name: mount-target-az1
  namespace: ack-system
spec:
  fileSystemRef:
    from:
      name: my-filesystem
      namespace: ack-system
  subnetID: subnet-02c1239a3f791d57f
  securityGroups:
    - sg-0d74b83661232689d

In both cases, even after I leave them for a day, the status is still not correctly updated.

[gnadaban]

I'm not sure I'd call this level of completeness "Generally Available" worthy.

[gnadaban]

It would seem that this only happens when encrypted: true is configured.

[gnadaban]

Other fields like policy or lifecyclePolicies have a similar effect: when configured, the resource never gets initialized.
Also, if initially the policy field is set, and is later deleted, the resource never gets synced again due to a missing field error.

{"level":"error","ts":"2024-10-03T20:44:44.492Z","msg":"Reconciler error","controller":"filesystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","FileSystem":{"name":"dummyfs","namespace":"dummy"},"namespace":"dummy","name":"dummyfs","reconcileID":"58c2ae6b-48a5-462d-aa9c-1837b2060f1c","error":"InvalidParameter: 1 validation error(s) found.\n- missing required field, PutFileSystemPolicyInput.Policy.\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}