Rolling update for multiple dependent resources of the same type

[moayad-alyaghshi]

Hi,

We have a reconciler which marks two different StatefulSets as dependent resources. The reconciler logic which we are looking for is that the first StatefulSet must be reconciled and its replica becomes ready before the second StatefulSets is reconciled.

We managed to achieve that for the create operation by defining a workflow and add the dependent resources of the StatefulSets to it, and add a ready postcondition for the dependent resource of the first StatefulSet which checks the number of ready replicas. Unfortunately, this did not work for the update scenario. The first StatefulSet was reconciled and updated first (the StatefulSet will be recreated) as wanted, but the ready postcondition returned true when it was supposed to return false since there should be no ready replica yet, and that triggered the reconciliation of the second StatefulSets immediately. It seems the status of the first StatefulSet was not updated yet when the condition was triggered. We tried to use the value current replicas of the first StatefulSet instead of ready replicas, but that didn't help also.

Do you have a solution for this issue, and are we going into the right direction or there is a better way to achieve the logic we're looking for?

Thank you!

[csviri]

Hi @moayad-alyaghshi , this is quite interesting, if you could provide a mininalistic implementation / sample, where it could be reproduced that would be a great help, and I could take a look check what is wrong.

What you describe should not be a problem in theory.

[moayad-alyaghshi]

I attached a sample project using Quarkus. The steps to reproduce the use case which we're trying to achieve:

1. Apply the file example.yaml
2. Two StatefulSets will be created sequentially with the version 1.16.0 (the replica StatefulSet will not be reconciled until the primary StatefulSet has a ready replica, which is the wanted behaviour)
3. Update the example.yaml with a different version (e.g. 1.17.0)
4. Both of the StatefulSets will be reconciled in parallel, so they will be terminated and recreated. This's not the expected behaviour, which should be the same as what happened in step 2

sample-operator.zip

[csviri]

1. There is nothing wrong in this case with you sample code was (as far I can see now). However, we don't state it in the docs - will add it there but informers for the same type should not be overlapping, would rather advice to use one informer for both dependent resources, see how it is done here:
   https://github.com/java-operator-sdk/java-operator-sdk/blob/c4fc7986a88b3ecb01ca1a067b32083a93eb2240/operator-framework/src/test/java/io/javaoperatorsdk/operator/sample/multiplemanageddependentsametype/MultipleManagedDependentResourceReconciler.java#L16-L57

[csviri]

2. On there hand if you patch the image on stateful set, the ready replicas does no change. Try to do this when you change a stateful set with kubectl:
   kubectl get statefulset web -o yaml --watch

(So set this is in one console and change the stateful set in an other, so you don't miss the updates)

It will show you how a stateful set changes.

Note that status part (this is after more changes):

status:
  availableReplicas: 2
  collisionCount: 0
  currentRevision: web-689c9989cf
  observedGeneration: 4
  readyReplicas: 2
  replicas: 3
  updateRevision: web-76589f55d9
  updatedReplicas: 3

You will see how the resource changes. In first few changes the controller of the StatefulSet does not change the ready replicas.
So the real problem here is with the logic of you condition implementation. First (not an expert on this), but you have to check if the metadata.generation is same as the status.observedGeneration in status. Then, probably you have to check if the currentRevision mathes the updatedRevision. It seems to me if those match, the rolling update is finished. But yes probably you should also then match if replicas equal to availableReplicas.

So pls try to update your condition according this logic (or better).

[moayad-alyaghshi]

1. I will modify the code to use one informer for both StatefulSets. The thing which I am still not sure about is what to use, managed or standalone mode. At the beginning when we had plain dependent resources without a workflow, it seemed that using standalone will give us more control (reconcile DRs depending on conditions), but now using a workflow it seems we can achieve the exact same functionality using the different modes.
2. You're right. The first thing which is going to be changed in the StatefulSet is metadata.generation, so comparing it to the status.observedGeneration is the right thing to do.

Thank you for the help. Much appreciated.