generalization of the update matcher #2014

shawkins · 2023-08-11T11:03:20Z

This generalizes the update and match logic for fields other than spec and metadata. This should make the usage of specific subclasses unnecessary.

shawkins · 2023-08-11T11:21:28Z

I should mention that both the current logic and this logic have problems with matching stuff in the desired state that ends up being pruned. This pr does make that problem worse - if the desired state for example includes the status subresource that will be seen as a difference is it doesn't match exactly with the actual. But I don't think users should generally be doing this.

shawkins · 2023-08-11T14:57:59Z

@metacosm @csviri there are two parts here

the first is to generalize the update logic. I did that initially with a custom mapper, but that complexity isn't needed we can work just over the regular map forms. We'll basically replace any top-level field in the actual with what is in the desired with special handling for the metadata. This is similar to the Jackson updateValue method, which we could consider exposing.
the second is to generalize the matching. Based upon all of the existing subclasses it makes the assumption that you want equality for anything on the desired that is non-spec field.

csviri

Looks, good just some comments, and missing unit tests :)

...vaoperatorsdk/operator/processing/dependent/kubernetes/GenericKubernetesResourceMatcher.java

...k/operator/processing/dependent/kubernetes/updatermatcher/GenericResourceUpdaterMatcher.java

csviri · 2023-08-14T08:45:04Z

...vaoperatorsdk/operator/processing/dependent/kubernetes/GenericKubernetesResourceMatcher.java

-    // https://github.com/fabric8io/kubernetes-client/issues/3816
-    var specDiffJsonPatch = getDiffsImpactingPathsWithPrefixes(wholeDiffJsonPatch, SPEC);
+    boolean specMatch = match(equality, wholeDiffJsonPatch, ignoreList, SPEC);
+    if (!specMatch) {


Please add unit tests at least for some of the special cases we were handling before?

Same applies here GenericKubernetesResourceMatcherTest covers all of the modifications.

csviri

LGTM

csviri · 2023-08-14T13:57:20Z

...vaoperatorsdk/operator/processing/dependent/kubernetes/GenericKubernetesResourceMatcher.java

@@ -24,6 +23,9 @@ public class GenericKubernetesResourceMatcher<R extends HasMetadata, P extends H
  private static final String OP = "op";
  public static final String METADATA_LABELS = "/metadata/labels";
  public static final String METADATA_ANNOTATIONS = "/metadata/annotations";
+  // without knowing the CRD we cannot ignore status as it may not be a subresource, so if it's
+  // included we expect it to match
+  private static Set<String> NOT_OTHER_FIELDS = Set.of(SPEC, "/metadata", "/apiVersion", "/kind");


Why SPEC is here? we want to match SPEC, or? I'm reading it wrong

also again would add here status, since that does not need to match I think even if it is not a subresource

Why SPEC is here? we want to match SPEC, or? I'm reading it wrong

spec and metadata have their own handling, so that's not covered by the "other fields" matching.

also again would add here status, since that does not need to match I think even if it is not a subresource

You would only want to do that if you suspect that users are including non-matching status that is not meaningful on their dependent resources. My guess would be that they are not including status, and if they are it would be because it's coming from the actual state (and would match), or that it's not a subresource and needs to be considered in matching. But if you feel strongly about it, it can be excluded.

spec and metadata have their own handling, so that's not covered by the "other fields" matching.

sorry totally missed that.

Yet another tweak would be to change the logic to just walk through the diffs once and do prefix matching to determine if the change is allowable that may make things even clearer.

You would only want to do that if you suspect that users are including non-matching status that is not meaningful on their dependent resources. My guess would be that they are not including status, and if they are it would be because it's coming from the actual state (and would match), or that it's not a subresource and needs to be considered in matching. But if you feel strongly about it, it can be excluded.

So these are resources which usually created, but not managed by the controller, so in normal case we apply the desired state using SSA, that is usually the spec. We don't care about the status at all at this phase, we might case in a postCondition, or when we are setting the status for the primary resource.

We don't care about the status at all at this phase,

If you assume it's a subresource. If not, then you do.

yep, true, but again that should not be a feature flag? we are comparing the desired with the actual, the actual would contain the status even if it is a subresource, but the desired won't, wouldn't this always fail to match in that case?

we are comparing the desired with the actual, the actual would contain the status even if it is a subresource, but the desired won't, wouldn't this always fail to match in that case?

Ah, the issue you are raising is with the true parameter with https://github.com/operator-framework/java-operator-sdk/pull/2014/files#diff-3425845ceca0cbb05292b87cdceec9960b4342cefac82c18ab929a5ef22e5288R211 - if someone leaves the status unpopulated in the desired (which is expected) the exact match here will cause a problem. I must have been implicitly assuming that all are add ops would apply here, but that would obviously require a separate case. Let me make another revision and add a test.

Further refactored to clarify the matching algorithm and added status as excluded.

shawkins · 2023-08-14T21:07:06Z

One last thought is whether you want something for the loss of the protected method updateClonedActual - a release note? Bring back a similar method, or just let people override updateResource after calling the super method?

csviri · 2023-08-15T06:54:45Z

One last thought is whether you want something for the loss of the protected method updateClonedActual - a release note? Bring back a similar method, or just let people override updateResource after calling the super method?

I think overriding updateResource is good enough for now, will see about feedback, and will make it easier if there will be a lot of buzz around it.

csviri

LGTM

also adding a test that removes a field not present in the desired

metacosm · 2023-08-23T08:38:07Z

...k/operator/processing/dependent/kubernetes/updatermatcher/GenericResourceUpdaterMatcher.java

-    var clonedActual = context.getControllerConfiguration().getConfigurationService()
-        .getResourceCloner().clone(actual);
+    KubernetesSerialization kubernetesSerialization =
+        context.getClient().getKubernetesSerialization();


I'm wondering if this is opening potential issues where the cloning occurs in a different manner than here if, for some reason, the configured resource cloner has a different serialization set up… Should we remove the cloner altogether?

If this was the only instance of the cloner being used, it could be removed or deprecated. There is also a clone method on KubernetesSerialization.

metacosm

LGTM apart from one comment.

* feat: create resource only if not exists (#2001) * feat: leader election callbacks (#2015) * discriminator improvements (#2013) * generalization of the update matcher * consolidating the diffs also adding a test that removes a field not present in the desired --------- Co-authored-by: Attila Mészáros <[email protected]>

* feat: create resource only if not exists (#2001) * feat: leader election callbacks (#2015) * discriminator improvements (#2013) * generalization of the update matcher * consolidating the diffs also adding a test that removes a field not present in the desired --------- Co-authored-by: Attila Mészáros <[email protected]> Signed-off-by: csviri <[email protected]>

* feat: create resource only if not exists (#2001) * feat: leader election callbacks (#2015) * discriminator improvements (#2013) * generalization of the update matcher * consolidating the diffs also adding a test that removes a field not present in the desired --------- Co-authored-by: Attila Mészáros <[email protected]> Signed-off-by: csviri <[email protected]> Signed-off-by: Attila Mészáros <[email protected]>

* feat: create resource only if not exists (operator-framework#2001) * feat: leader election callbacks (operator-framework#2015) * discriminator improvements (operator-framework#2013) * generalization of the update matcher * consolidating the diffs also adding a test that removes a field not present in the desired --------- Co-authored-by: Attila Mészáros <[email protected]> Signed-off-by: csviri <[email protected]> Signed-off-by: Attila Mészáros <[email protected]> Signed-off-by: Steve Hawkins <[email protected]>

* feat: create resource only if not exists (#2001) * feat: leader election callbacks (#2015) * discriminator improvements (#2013) * generalization of the update matcher * consolidating the diffs also adding a test that removes a field not present in the desired --------- Co-authored-by: Attila Mészáros <[email protected]> Signed-off-by: csviri <[email protected]> Signed-off-by: Attila Mészáros <[email protected]>

openshift-ci bot requested review from adam-sandor and andreaTP August 11, 2023 11:03

shawkins marked this pull request as draft August 11, 2023 11:03

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 11, 2023

shawkins force-pushed the generalized_match branch 2 times, most recently from 69d0048 to 97da6cf Compare August 11, 2023 14:32

shawkins changed the title ~~draft of a generalization of the update matcher~~ generalization of the update matcher Aug 11, 2023

shawkins marked this pull request as ready for review August 11, 2023 14:33

openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 11, 2023

openshift-ci bot requested a review from csviri August 11, 2023 14:33

shawkins force-pushed the generalized_match branch from 97da6cf to 243b320 Compare August 11, 2023 14:52

shawkins force-pushed the generalized_match branch 2 times, most recently from 0e5b9cf to 2fbd8a0 Compare August 12, 2023 11:50

csviri requested changes Aug 14, 2023

View reviewed changes

shawkins force-pushed the generalized_match branch 2 times, most recently from 595f8d2 to 7bd9674 Compare August 14, 2023 12:10

shawkins requested a review from csviri August 14, 2023 13:33

csviri approved these changes Aug 14, 2023

View reviewed changes

csviri reviewed Aug 14, 2023

View reviewed changes

shawkins force-pushed the generalized_match branch from 54a65a6 to 1d53df0 Compare August 14, 2023 16:11

csviri approved these changes Aug 15, 2023

View reviewed changes

csviri added 3 commits August 15, 2023 14:04

feat: create resource only if not exists (operator-framework#2001)

d5a5737

feat: leader election callbacks (operator-framework#2015)

ffdcc10

discriminator improvements (operator-framework#2013)

5c458e5

csviri force-pushed the next branch from d7b5b87 to 5c458e5 Compare August 15, 2023 12:04

generalization of the update matcher

8675f3f

consolidating the diffs

0357ac3

also adding a test that removes a field not present in the desired

shawkins force-pushed the generalized_match branch from 1d53df0 to 0357ac3 Compare August 15, 2023 12:31

csviri force-pushed the next branch from 642786b to f03ebb8 Compare August 22, 2023 10:28

Merge branch 'next' into generalized_match

4de80dd

metacosm reviewed Aug 23, 2023

View reviewed changes

metacosm self-requested a review August 23, 2023 08:43

metacosm approved these changes Aug 23, 2023

View reviewed changes

Merge branch 'next' into generalized_match

fe720ec

metacosm mentioned this pull request Aug 23, 2023

Consider removing Cloner interface #2027

Open

csviri merged commit 6839b63 into operator-framework:next Aug 28, 2023
17 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

generalization of the update matcher #2014

generalization of the update matcher #2014

shawkins commented Aug 11, 2023

shawkins commented Aug 11, 2023

shawkins commented Aug 11, 2023

csviri left a comment

csviri Aug 14, 2023

shawkins Aug 14, 2023

csviri left a comment

csviri Aug 14, 2023

csviri Aug 14, 2023

shawkins Aug 14, 2023

csviri Aug 14, 2023

shawkins Aug 14, 2023 •

edited

Loading

csviri Aug 14, 2023

shawkins Aug 14, 2023

csviri Aug 14, 2023

shawkins Aug 14, 2023

shawkins Aug 14, 2023

shawkins commented Aug 14, 2023

csviri commented Aug 15, 2023

csviri left a comment

metacosm Aug 23, 2023

shawkins Aug 23, 2023

metacosm left a comment

generalization of the update matcher #2014

generalization of the update matcher #2014

Conversation

shawkins commented Aug 11, 2023

shawkins commented Aug 11, 2023

shawkins commented Aug 11, 2023

csviri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csviri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawkins Aug 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shawkins commented Aug 14, 2023

csviri commented Aug 15, 2023

csviri left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

metacosm left a comment

Choose a reason for hiding this comment

shawkins Aug 14, 2023 •

edited

Loading