CSO status become false in upgrade from 2.10.1 to 2.11.0

[saurabhwani5]

Describe the bug

When tried to upgrade from 2.10.1 to 2.11.0 following error is coming Synchronization of node/driver ibm-spectrum-scale-csi DaemonSet failed for the CSISCaleOperator instance ibm-spectrum-scale-csi
CSO status :

[root@csi-kube129-1-x-master ~]# oc get cso -w
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.1    True
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    False
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    True

How to Reproduce?

Please list the steps to help development teams reproduce the behavior

1. Install CSI 2.10.1:

[root@csi-kube129-1-x-master ~]# oc get cso -w
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.1    True

2. Create PVC of version 2 :

[root@csi-kube129-1-x-master ~]# oc get pvc
NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                     VOLUMEATTRIBUTESCLASS   AGE
ibm-spectrum-scale-pvc-clone-from-pvc   Bound    pvc-e868506e-245a-4160-bda2-5030c867c6b4   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 24m
ibm-spectrum-scale-pvc-from-snapshot    Bound    pvc-a405e6a9-7eae-4907-874e-75b5b3f6b8f2   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 25m
scale-advance-pvc-1                     Bound    pvc-be292e11-0017-460f-9671-7eda17a60bab   1Gi        RWX            ibm-spectrum-scale-csi-advance   <unset>                 33m

3. Upgrade to 2.11.0 :

[root@csi-kube129-1-x-master saurabh]# oc apply -f ibm-spectrum-scale-csi-operator.yaml
deployment.apps/ibm-spectrum-scale-csi-operator configured
clusterrole.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator configured
clusterrolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator unchanged
serviceaccount/ibm-spectrum-scale-csi-operator unchanged
customresourcedefinition.apiextensions.k8s.io/csiscaleoperators.csi.ibm.com configured

4. Check cso status

[root@csi-kube129-1-x-master ~]# oc get cso -w
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.1    True
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    False
ibm-spectrum-scale-csi   2.11.0    True
ibm-spectrum-scale-csi   2.11.0    True

5. Check CSO description:

[root@csi-kube129-1-x-master ~]# oc describe cso
Name:         ibm-spectrum-scale-csi
Namespace:    ibm-spectrum-scale-csi-driver
Labels:       app.kubernetes.io/instance=ibm-spectrum-scale-csi-operator
              app.kubernetes.io/managed-by=ibm-spectrum-scale-csi-operator
              app.kubernetes.io/name=ibm-spectrum-scale-csi-operator
              release=ibm-spectrum-scale-csi-operator
Annotations:  <none>
API Version:  csi.ibm.com/v1
Kind:         CSIScaleOperator
Metadata:
  Creation Timestamp:  2024-04-24T01:42:35Z
  Finalizers:
    finalizer.csiscaleoperators.csi.ibm.com
  Generation:        2
  Resource Version:  6221078
  UID:               11932dfe-4d32-4b73-814f-a7f80d31268a
Spec:
  Attacher Node Selector:
    Key:    scale
    Value:  true
  Clusters:
    Id:  453210684566980716
    Primary:
      Primary Fs:  fs1
    Rest API:
      Gui Host:       10.11.107.129
    Secrets:          guisecret
    Secure Ssl Mode:  false
    Id:               4181049054023231843
    Rest API:
      Gui Host:              10.11.57.54
    Secrets:                 remotenewsecret
    Secure Ssl Mode:         false
  Consistency Group Prefix:  4500ce03-4b53-4a3d-b884-ba5f69868521
  Plugin Node Selector:
    Key:    scale
    Value:  true
  Provisioner Node Selector:
    Key:    scale
    Value:  true
  Resizer Node Selector:
    Key:    scale
    Value:  true
  Snapshotter Node Selector:
    Key:    scale
    Value:  true
Status:
  Conditions:
    Last Transition Time:  2024-04-24T02:21:17Z
    Message:               The CSI driver resources have been created/updated successfully
    Reason:                CSIConfigured
    Status:                True
    Type:                  Success
  Versions:
    Name:     ibm-spectrum-scale-csi
    Version:  2.11.0
Events:
  Type     Reason         Age                From              Message
  ----     ------         ----               ----              -------
  Normal   CSIConfigured  37m (x4 over 38m)  CSIScaleOperator  The CSI driver resources have been created/updated successfully
  Warning  UpdateFailed   29s                CSIScaleOperator  Synchronization of ibm-spectrum-scale-csi-resizer Deployment failed for the CSISCaleOperator instance ibm-spectrum-scale-csi
  Normal   CSIConfigured  15s (x4 over 33s)  CSIScaleOperator  The CSI driver resources have been created/updated successfully

Expected behavior

CSO should not give any error log

Logs

/scale-csi/D.1137
csisnap.tar.gz

[deeghuge]

@saurabhwani5 do keep eye on this during current upgrade tests