When the ConfigMap is deleted, the CSO status temporarily switches to 'false' for a few seconds

[saurabhwani5]

Describe the bug

When the Configmap is deleted then CSO status goes to false for few seconds

How to Reproduce?

1. Install CSI 2.9.0 :

[root@OCP pr1056]# oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-csi-attacher-64fb7cf8d9-8jlh9      1/1     Running   0          3s
ibm-spectrum-scale-csi-attacher-64fb7cf8d9-j48p4      1/1     Running   0          3s
ibm-spectrum-scale-csi-cg94s                          3/3     Running   0          3s
ibm-spectrum-scale-csi-nf2vz                          3/3     Running   0          3s
ibm-spectrum-scale-csi-operator-747755cc86-sqdwp      1/1     Running   0          112s
ibm-spectrum-scale-csi-provisioner-7df5887df9-l78wn   1/1     Running   0          3s
ibm-spectrum-scale-csi-resizer-7c9c4ff4bc-2gdxh       1/1     Running   0          3s
ibm-spectrum-scale-csi-snapshotter-74d845c99b-5d6sz   1/1     Running   0          3s
ibm-spectrum-scale-csi-sq7tq                          3/3     Running   0          3s
[root@OCP pr1056]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.9.0     True

2. Create Configmap as following :

[root@OCP pr1056]# cat cm.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: ibm-spectrum-scale-csi-config
  namespace: ibm-spectrum-scale-csi
data:
  VAR_DRIVER_LOGLEVEL: TRACE
  VAR_DRIVER_PERSISTENT_LOG: ENABLED
  VAR_DRIVER_VOLUME_STATS_CAPABILITY: DISABLED
  VAR_DRIVER_NODEPUBLISH_METHOD: SYMLINK
  DRIVER_UPGRADE_MAXUNAVAILABLE: 90%
  VAR_DRIVER_DISCOVER_CG_FILESET: ENABLED
[root@OCP pr1056]# oc apply -f cm.yaml
configmap/ibm-spectrum-scale-csi-config created

3. Upgrade to CSI 2.10 with following Images:

Operator : quay.io/badri_pathak/ibm-spectrum-scale-csi-operator:upgrade_events_failed_v1
Driver: quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:0c0344663aba81eda14ed01118bd3e59fee9c1a258b3b9f360e70221a226af5d
[root@OCP pr1056]# oc edit deployment ibm-spectrum-scale-csi-operator
deployment.apps/ibm-spectrum-scale-csi-operator edited
[root@OCP pr1056]# oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-csi-7krjq                          3/3     Running   0          20s
ibm-spectrum-scale-csi-9ltvk                          3/3     Running   0          20s
ibm-spectrum-scale-csi-attacher-775c787cd7-mtz5p      1/1     Running   0          21s
ibm-spectrum-scale-csi-attacher-775c787cd7-nws55      1/1     Running   0          21s
ibm-spectrum-scale-csi-operator-54485c77d6-zhtln      1/1     Running   0          43s
ibm-spectrum-scale-csi-provisioner-74dc9dff59-72cp4   1/1     Running   0          21s
ibm-spectrum-scale-csi-resizer-78f7684fff-xndd9       1/1     Running   0          21s
ibm-spectrum-scale-csi-snapshotter-5f77874594-dm66v   1/1     Running   0          21s
ibm-spectrum-scale-csi-t7rgb                          3/3     Running   0          20s
[root@OCP pr1056]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.0    True

4. Delete configmap when all pods are up :

[root@OCP pr1056]# oc delete cm ibm-spectrum-scale-csi-config
configmap "ibm-spectrum-scale-csi-config" deleted

5. Check the CSO status simultaneously :

[root@OCP pr1056]# oc get cso -w
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.9.0     True
ibm-spectrum-scale-csi   2.10.0    True
ibm-spectrum-scale-csi   2.10.0    True
ibm-spectrum-scale-csi   2.10.0    True
ibm-spectrum-scale-csi   2.10.0    False
ibm-spectrum-scale-csi   2.10.0    True
ibm-spectrum-scale-csi   2.10.0    True

6. Description of CSO :

Events:
  Type     Reason         Age                     From              Message
  ----     ------         ----                    ----              -------
  Warning  UpdateFailed   5m23s                   CSIScaleOperator  Synchronization of node/driver ibm-spectrum-scale-csi DaemonSet failed for the CSISCaleOperator instance ibm-spectrum-scale-csi
  Normal   CSIConfigured  5m20s (x14 over 5m56s)  CSIScaleOperator  The CSI driver resources have been created/updated successfully
  Warning  UpdateFailed   39s                     CSIScaleOperator  Synchronization of ibm-spectrum-scale-csi-attacher Deployment failed for the CSISCaleOperator instance ibm-spectrum-scale-csi
  Normal   CSIConfigured  36s (x5 over 2m59s)     CSIScaleOperator  The CSI driver resources have been created/updated successfully

Expected behavior

CSO status should be true when cm is deleted

Logs

scale-csi/D.1059
mustgather.tar.gz