[Juju 2.9.44] Charm state is not being passed down to Juju Controller

[moisesbenzan]

kubeflow-volumues is stuck in maintenance mode, as seen from the Juju status, but the debug logs for the charm indicate that no operation is running and the unit is indeed idle.

kubeflow-roles                                      active           1  kubeflow-roles           1.7/stable      113  10.152.183.205  no       
kubeflow-volumes           res:oci-image@d261609    maintenance      1  kubeflow-volumes         1.7/stable      178  10.152.183.21   no       Setting pod spec

We suspect the charm is failing to report the state back to Juju, hence Juju thinks it is still under maintenance.

Link to run artifacts can be found here: https://oil-jenkins.canonical.com/artifacts/28b8d549-4379-43f7-84bb-b8fa30ba19ee/index.html
Pod logs here: https://oil-jenkins.canonical.com/artifacts/28b8d549-4379-43f7-84bb-b8fa30ba19ee/generated/generated/juju_kubernetes_controller/juju-crashdump-controller-2023-07-16-09.30.20.tar.gz

[i-chvets]

@moisesbenzan What Kubernets is being used for this deployment?

[DnPlas]

Thanks for reporting this @moisesbenzan, I have a suspicion that the underlying charmed kubernetes storage configuration is messing up with the kubeflow-volumes charm.

We suspect the charm is failing to report the state back to Juju, hence Juju thinks it is still under maintenance.

Our charm flow is as follows:

1. Set MainentanceStatus with message Setting pod spec L58
2. Call self.model.pod.set_spec(..) L60
3. Set ActiveStatus L127

So in fact something is happening between 1 and 3, causing the unit state to not change.

Unfortunately from the logs I could not find out what's wrong, in fact the juju debug-logs are truncated. If you could provide the output of juju debug-logs --replay that would be helpful, as well as the output of kubectl describe all -A, or at least describing the kubeflow-volume pods (I say pods because there are two different ones).
Finally, please share with us how the storage in your charmed kubernetes deployment is configured.

[orfeas-k]

Closing this since no information has been given in order to track down the issue. Feel free to re-open if this occurs again and/or more information is available.

[orfeas-k]

On top of that, we are rewriting the charm this pulse AFAIK. Thus, issues like this could cease to exist.