Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NullPointerException in SSABasedGenericKubernetesResourceMatcher #2032

Closed
kosmoz opened this issue Aug 24, 2023 · 5 comments · Fixed by #2042 · May be fixed by #2037
Closed

NullPointerException in SSABasedGenericKubernetesResourceMatcher #2032

kosmoz opened this issue Aug 24, 2023 · 5 comments · Fixed by #2042 · May be fixed by #2037
Assignees
@csviri

Comments

@kosmoz
Copy link

kosmoz commented Aug 24, 2023

Bug Report

What did you do?

We ran into this issue when using unsupported beta features on an older Kubernetes version:

  • Create a StatefulSet dependent resource with persistentVolumeClaimRetentionPolicy on a cluster running Kubernetes version 1.26.
  • Kubernetes adds persistentVolumeClaimRetentionPolicy to the managedFields but ignores the value
  • When getting the StatefulSet, the SDK expects all managed fields to be present in the resource but in this case, persistentVolumeClaimRetentionPolicy is missing.

What did you expect to see?

At most, a warning should be printed.

What did you see instead? Under which circumstances?

Error during event processing ExecutionScope{ resource id: ResourceID{name='redacted', namespace='redacted'}, version: 8966444451} failed. 
io.javaoperatorsdk.operator.AggregatedOperatorException: Exception(s) during workflow execution. Details:
 - eu.glasskube.operator.apps.vault.dependent.VaultStatefulSet -> java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because "actualMap" is null
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:144)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.fillResultsAndTraverseFurther(SSABasedGenericKubernetesResourceMatcher.java:166)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:138)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.fillResultsAndTraverseFurther(SSABasedGenericKubernetesResourceMatcher.java:166)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:138)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.matches(SSABasedGenericKubernetesResourceMatcher.java:90)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.match(KubernetesDependentResource.java:169)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.match(KubernetesDependentResource.java:32)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:67)
	at io.javaoperatorsdk.operator.processing.dependent.SingleDependentResourceReconciler.reconcile(SingleDependentResourceReconciler.java:19)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:52)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileExecutor$NodeReconcileExecutor.doRun(WorkflowReconcileExecutor.java:115)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.NodeExecutor.run(NodeExecutor.java:22)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowResult.throwAggregateExceptionIfErrorsPresent(WorkflowResult.java:41)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileResult.throwAggregateExceptionIfErrorsPresent(WorkflowReconcileResult.java:9)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.DefaultWorkflow.reconcile(DefaultWorkflow.java:95)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:110)
	at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219)
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:109)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:140)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:121)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:91)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:409)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Environment

Kubernetes cluster type:

vanilla

$ Mention java-operator-sdk version from pom.xml file

4.4.2

$ java -version

openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7, mixed mode, sharing)

$ kubectl version

Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.4
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1

Possible Solution

Check if actualMap is null before adding to the result.

Additional context

Related: glasskube/operator#267
@shawkins
Copy link
Collaborator

Relates to #2028 - it's the same NPE that happens when using a secret with stringData in the desired state.

@csviri
Copy link
Collaborator

csviri commented Aug 25, 2023

Hi @kosmoz , on short notice you might want to try the generic matcher, see:
https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/GenericKubernetesResourceMatcher.java#L19-L19

also:
https://javaoperatorsdk.io/docs/v4-4-migration#using-server-side-apply-in-dependent-resources

Would be great if you could provide a simple project to reproduce. Or provide the logs from here: https://github.com/shawkins/java-operator-sdk/blob/c879806ba279c2a285639344a7e34c01118c8c01/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/SSABasedGenericKubernetesResourceMatcher.java#L86-L86

thank you!

@csviri csviri self-assigned this Aug 25, 2023
@kosmoz
Copy link
Author

kosmoz commented Aug 25, 2023 •
edited
Loading

Hi! I'm not sure if I will have time to create a minimal reprodicution repo. However the steps would be pretty straight forward:

  1. create an operator with one controller for a CRD
  2. create a managed dependent resource for that CRD. This should be a StatefulSet where persistentVolumeClaimRetentionPolicy is set to something other than null.
  3. On a cluster running a Kubernetes version <1.27 create a resource for that CRD

Maybe I will have time next week, but in the meantime, here are the logs (I pruned them to the lines that I think matter for this error).

2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] TRACE i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - Original actual: 
 StatefulSet(apiVersion=apps/v1, kind=StatefulSet, metadata=ObjectMeta(annotations={}, creationTimestamp=2023-08-25T09:00:45Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[ManagedFieldsEntry(apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1=FieldsV1(additionalProperties={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}}, f:spec={f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}}}), manager=vaultreconciler, operation=Apply, subresource=null, time=2023-08-25T09:00:45Z, additionalProperties={}), ManagedFieldsEntry(apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1=FieldsV1(additionalProperties={f:status={f:collisionCount={}, f:currentReplicas={}, f:currentRevision={}, f:observedGeneration={}, f:replicas={}, f:updateRevision={}, f:updatedReplicas={}}}), manager=kube-controller-manager, operation=Update, subresource=status, time=2023-08-25T09:00:45Z, additionalProperties={})], name=vault-test, namespace=default, ownerReferences=[OwnerReference(apiVersion=glasskube.eu/v1alpha1, blockOwnerDeletion=null, controller=null, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9, additionalProperties={})], resourceVersion=2242, selfLink=null, uid=33102e14-0069-4979-8739-64f86f00822e, additionalProperties={}), spec=StatefulSetSpec(minReadySeconds=null, ordinals=null, persistentVolumeClaimRetentionPolicy=null, podManagementPolicy=OrderedReady, replicas=1, revisionHistoryLimit=10, selector=LabelSelector(matchExpressions=[], matchLabels={app.kubernetes.io/instance=test, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault}, additionalProperties={}), serviceName=vault-test-headless, template=PodTemplateSpec(metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[], name=null, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[EnvVar(name=SKIP_SETCAP, value=true, valueFrom=null, additionalProperties={}), EnvVar(name=HOSTNAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.name, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=SUBDOMAIN, value=vault-test-headless, valueFrom=null, additionalProperties={}), EnvVar(name=POD_IP, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=status.podIP, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_API_ADDR, value=https://$(POD_IP):8200, valueFrom=null, additionalProperties={}), EnvVar(name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201, valueFrom=null, additionalProperties={}), EnvVar(name=DB_HOST, value=vault-test-db-rw, valueFrom=null, additionalProperties={}), EnvVar(name=DB_NAME, value=vault, valueFrom=null, additionalProperties={}), EnvVar(name=DB_USERNAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=null, resourceFieldRef=null, secretKeyRef=SecretKeySelector(key=username, name=vault-test-db-app, optional=null, additionalProperties={}), additionalProperties={}), additionalProperties={}), EnvVar(name=DB_PASSWORD, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=null, resourceFieldRef=null, secretKeyRef=SecretKeySelector(key=password, name=vault-test-db-app, optional=null, additionalProperties={}), additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME), valueFrom=null, additionalProperties={}), EnvVar(name=VAULT_K8S_POD_NAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.name, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_K8S_NAMESPACE, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.namespace, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={})], envFrom=[], image=hashicorp/vault:1.14.0, imagePullPolicy=IfNotPresent, lifecycle=Lifecycle(postStart=null, preStop=LifecycleHandler(exec=ExecAction(command=[killall, vault], additionalProperties={}), httpGet=null, tcpSocket=null, additionalProperties={}), additionalProperties={}), livenessProbe=null, name=vault, ports=[ContainerPort(containerPort=8200, hostIP=null, hostPort=null, name=https, protocol=TCP, additionalProperties={}), ContainerPort(containerPort=8201, hostIP=null, hostPort=null, name=https-internal, protocol=TCP, additionalProperties={})], readinessProbe=null, resources=ResourceRequirements(claims=[], limits={memory=100Mi}, requests={memory=30Mi}, additionalProperties={}), securityContext=SecurityContext(allowPrivilegeEscalation=false, capabilities=Capabilities(add=[], drop=[ALL], additionalProperties={}), privileged=null, procMount=null, readOnlyRootFilesystem=true, runAsGroup=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, seccompProfile=null, windowsOptions=null, additionalProperties={}), startupProbe=null, stdin=null, stdinOnce=null, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, tty=null, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=/glasskube/config, mountPropagation=null, name=config, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/glasskube/tls, mountPropagation=null, name=tls, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/vault/audit, mountPropagation=null, name=audit, readOnly=null, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsConfig=null, dnsPolicy=ClusterFirst, enableServiceLinks=null, ephemeralContainers=[], hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostUsers=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, os=null, overhead={}, preemptionPolicy=null, priority=null, priorityClassName=null, readinessGates=[], resourceClaims=[], restartPolicy=Always, runtimeClassName=null, schedulerName=default-scheduler, schedulingGates=[], securityContext=PodSecurityContext(fsGroup=1000, fsGroupChangePolicy=null, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000, seLinuxOptions=null, seccompProfile=null, supplementalGroups=[], sysctls=[], windowsOptions=null, additionalProperties={}), serviceAccount=vault-test, serviceAccountName=vault-test, setHostnameAsFQDN=null, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=10, tolerations=[], topologySpreadConstraints=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=ConfigMapVolumeSource(defaultMode=420, items=[], name=vault-test, optional=null, additionalProperties={}), csi=null, downwardAPI=null, emptyDir=null, ephemeral=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=config, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=null, ephemeral=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=tls, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=SecretVolumeSource(defaultMode=420, items=[], optional=null, secretName=vault-test-tls, additionalProperties={}), storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), additionalProperties={}), updateStrategy=StatefulSetUpdateStrategy(rollingUpdate=RollingUpdateStatefulSetStrategy(maxUnavailable=null, partition=0, additionalProperties={}), type=RollingUpdate, additionalProperties={}), volumeClaimTemplates=[PersistentVolumeClaim(apiVersion=v1, kind=PersistentVolumeClaim, metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={}, managedFields=[], name=audit, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PersistentVolumeClaimSpec(accessModes=[ReadWriteOnce], dataSource=null, dataSourceRef=null, resources=ResourceRequirements(claims=[], limits={}, requests={storage=1Gi}, additionalProperties={}), selector=null, storageClassName=null, volumeMode=Filesystem, volumeName=null, additionalProperties={}), status=PersistentVolumeClaimStatus(accessModes=[], allocatedResources={}, capacity={}, conditions=[], phase=Pending, resizeStatus=null, additionalProperties={}), additionalProperties={})], additionalProperties={}), status=StatefulSetStatus(availableReplicas=0, collisionCount=0, conditions=[], currentReplicas=1, currentRevision=vault-test-5bd5455664, observedGeneration=1, readyReplicas=null, replicas=1, updateRevision=vault-test-5bd5455664, updatedReplicas=1, additionalProperties={}), additionalProperties={}) 
 original desired: 
 {apiVersion=apps/v1, kind=StatefulSet, metadata={labels={app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/version=1.14.0, app.kubernetes.io/part-of=vault}, name=vault-test, namespace=default, ownerReferences=[{apiVersion=glasskube.eu/v1alpha1, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9}]}, spec={persistentVolumeClaimRetentionPolicy={whenDeleted=Delete, whenScaled=Retain}, replicas=1, selector={matchLabels={app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/part-of=vault}}, serviceName=vault-test-headless, template={metadata={labels={app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/version=1.14.0, app.kubernetes.io/part-of=vault}}, spec={containers=[{args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[{name=SKIP_SETCAP, value=true}, {name=HOSTNAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=SUBDOMAIN, value=vault-test-headless}, {name=POD_IP, valueFrom={fieldRef={apiVersion=v1, fieldPath=status.podIP}}}, {name=VAULT_API_ADDR, value=https://$(POD_IP):8200}, {name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201}, {name=DB_HOST, value=vault-test-db-rw}, {name=DB_NAME, value=vault}, {name=DB_USERNAME, valueFrom={secretKeyRef={key=username, name=vault-test-db-app}}}, {name=DB_PASSWORD, valueFrom={secretKeyRef={key=password, name=vault-test-db-app}}}, {name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME)}, {name=VAULT_K8S_POD_NAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=VAULT_K8S_NAMESPACE, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.namespace}}}], image=hashicorp/vault:1.14.0, lifecycle={preStop={exec={command=[killall, vault]}}}, name=vault, ports=[{containerPort=8200, name=https}, {containerPort=8201, name=https-internal}], resources={limits={memory=100Mi}, requests={memory=30Mi}}, securityContext={allowPrivilegeEscalation=false, capabilities={drop=[ALL]}, readOnlyRootFilesystem=true}, volumeMounts=[{mountPath=/glasskube/config, name=config, readOnly=true}, {mountPath=/glasskube/tls, name=tls, readOnly=true}, {mountPath=/vault/audit, name=audit}]}], securityContext={fsGroup=1000, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000}, serviceAccountName=vault-test, terminationGracePeriodSeconds=10, volumes=[{configMap={name=vault-test}, name=config}, {name=tls, secret={secretName=vault-test-tls}}]}}, volumeClaimTemplates=[{apiVersion=v1, kind=PersistentVolumeClaim, metadata={name=audit}, spec={accessModes=[ReadWriteOnce], resources={requests={storage=1Gi}}}}]}}  
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: metadata actual map value: {creationTimestamp=2023-08-25T09:00:45Z, generation=1, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[{apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}}, f:spec={f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}}}, manager=vaultreconciler, operation=Apply, time=2023-08-25T09:00:45Z}, {apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1={f:status={f:collisionCount={}, f:currentReplicas={}, f:currentRevision={}, f:observedGeneration={}, f:replicas={}, f:updateRevision={}, f:updatedReplicas={}}}, manager=kube-controller-manager, operation=Update, subresource=status, time=2023-08-25T09:00:45Z}], name=vault-test, namespace=default, ownerReferences=[{apiVersion=glasskube.eu/v1alpha1, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9}], resourceVersion=2242, uid=33102e14-0069-4979-8739-64f86f00822e} managedFieldValue: {f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}} 
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: labels actual map value: {app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0} managedFieldValue: {f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}} 
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: spec actual map value: {podManagementPolicy=OrderedReady, replicas=1, revisionHistoryLimit=10, selector={matchLabels={app.kubernetes.io/instance=test, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault}}, serviceName=vault-test-headless, template={metadata={labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}}, spec={containers=[{args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[{name=SKIP_SETCAP, value=true}, {name=HOSTNAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=SUBDOMAIN, value=vault-test-headless}, {name=POD_IP, valueFrom={fieldRef={apiVersion=v1, fieldPath=status.podIP}}}, {name=VAULT_API_ADDR, value=https://$(POD_IP):8200}, {name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201}, {name=DB_HOST, value=vault-test-db-rw}, {name=DB_NAME, value=vault}, {name=DB_USERNAME, valueFrom={secretKeyRef={key=username, name=vault-test-db-app}}}, {name=DB_PASSWORD, valueFrom={secretKeyRef={key=password, name=vault-test-db-app}}}, {name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME)}, {name=VAULT_K8S_POD_NAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=VAULT_K8S_NAMESPACE, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.namespace}}}], image=hashicorp/vault:1.14.0, imagePullPolicy=IfNotPresent, lifecycle={preStop={exec={command=[killall, vault]}}}, name=vault, ports=[{containerPort=8200, name=https, protocol=TCP}, {containerPort=8201, name=https-internal, protocol=TCP}], resources={limits={memory=100Mi}, requests={memory=30Mi}}, securityContext={allowPrivilegeEscalation=false, capabilities={drop=[ALL]}, readOnlyRootFilesystem=true}, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, volumeMounts=[{mountPath=/glasskube/config, name=config, readOnly=true}, {mountPath=/glasskube/tls, name=tls, readOnly=true}, {mountPath=/vault/audit, name=audit}]}], dnsPolicy=ClusterFirst, restartPolicy=Always, schedulerName=default-scheduler, securityContext={fsGroup=1000, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000}, serviceAccount=vault-test, serviceAccountName=vault-test, terminationGracePeriodSeconds=10, volumes=[{configMap={defaultMode=420, name=vault-test}, name=config}, {name=tls, secret={defaultMode=420, secretName=vault-test-tls}}]}}, updateStrategy={rollingUpdate={partition=0}, type=RollingUpdate}, volumeClaimTemplates=[{apiVersion=v1, kind=PersistentVolumeClaim, metadata={name=audit}, spec={accessModes=[ReadWriteOnce], resources={requests={storage=1Gi}}, volumeMode=Filesystem}, status={phase=Pending}}]} managedFieldValue: {f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}} 
2023-08-25T11:01:36.566+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: persistentVolumeClaimRetentionPolicy actual map value: null managedFieldValue: {f:whenDeleted={}, f:whenScaled={}}

Notice how in the last line, it says actual map value: null. This is not something keepOnlyManagedFields is designed to deal with, so that's where the error happens.

Side note: This can probably happen for other resources with unsupported properties as well. I don't understand why Kubernetes keeps the managed field but discards the property…

Thanks for your suggested workaround, we already removed the offending statement, since we want to support version 1.26 anyways. I just thought that the SDK could handle this edge-case more gracefully 🙂

Edit: For testing I created a cluster like this:

minikube start --profile=legacy --kubernetes-version 1.26

@shawkins
Copy link
Collaborator

Side note: This can probably happen for other resources with unsupported properties as well. I don't understand why Kubernetes keeps the managed field but discards the property…

I believe that case is happening as well here: kubernetes/kubernetes#118519 (comment)

on short notice you might want to try the generic matcher

While you won't get the NPE the matching won't work for pruned fields - the match logic will see that desired state has something that is missing in the actual.

@csviri csviri linked a pull request Aug 29, 2023 that will close this issue
reproduce ssa issue with persistentVolumeClaimRetentionPolicy #2037
Draft
@csviri csviri mentioned this issue Aug 29, 2023
Closed
@csviri
Copy link
Collaborator

csviri commented Aug 29, 2023

see #2038

@csviri csviri linked a pull request Sep 14, 2023 that will close this issue
feat: sanitization of resources for matching #2042
Merged
@csviri csviri closed this as completed Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@csviri csviri

Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@shawkins @csviri @kosmoz