Using a ConfigMap instead of a custom resource

[stefanbethke]

In some projects, we need to run in a cluster that is managed centrally in a multi-tenant setup. In particular, we cannot install anything cluster wide, including CRDs. (As an aside, even if we could install our CRD, what do we do if multiple tenants want to use the operator, but need different versions of the CRD?)

At least for my operator, it would be perfectly possible to have both the spec and the status as keys in a ConfigMap, and work off that. I've looked a little bit, but I didn't see an obvious way to create a Reconciler for a ConfigMap with some additional filtering attribute like a certain label or annotation. Is that already possible, and I missed it? Or do I simply create a Reconciler<ConfigMap> and do the filtering myself in reconcile()? If so, how can I switch between the CRD and the ConfigMap setup in Spring?

[stefanbethke]

So just looking at @ControllerConfiguration, I see the optional labelSelector attribute... I need to play with that. If it is indeed easily possible to do this, it would be fantastic to have this use case explained in the docs and have one of the examples extended to show how to do this.

[metacosm]

Yes, the labelSelector is there exactly for that purpose.

[csviri]

Hi @stefanbethke , it is touched in the docs, see:
https://javaoperatorsdk.io/docs/features
Section Support for Well Known (non-custom) Kubernetes Resources
So ConfigMap reconciliation is supported.

we cannot install anything cluster wide, including CRDs

Not sure if understand this, CRD is always cluster scoped.

As an aside, even if we could install our CRD, what do we do if multiple tenants want to use the operator, but need different versions of the CRD?

See versions in CRD: https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/#specify-multiple-versions

ConfigMap with some additional filtering attribute like a certain label or annotation. Reconciler for a ConfigMap with some additional filtering attribute like a certain label or annotation. Is that already possible, and I missed it? Or do I simply create a Reconciler and do the filtering myself in reconcile()?

You can use standard label selectors from Kuberentes. Can be specified in @ControllerConfiguration annotation for example.

[csviri]

At least for my operator, it would be perfectly possible to have both the spec and the status as keys in a ConfigMap

I would not advice to do this, if you have a different custom resources not a different version for an other tenant, just use a different Kind of API. Using ConfigMap instead of Custom Resources you looks functionalities, like generation awareness. So ConfigMap should be the last resort.

[metacosm]

In some projects, we need to run in a cluster that is managed centrally in a multi-tenant setup. In particular, we cannot install anything cluster wide, including CRDs.

Not being able to install a CRD is indeed a serious impediment to using operators. I wished you could install CRD per namespace…

(As an aside, even if we could install our CRD, what do we do if multiple tenants want to use the operator, but need different versions of the CRD?)

The CRD can define several versions, that shouldn't be an issue. Regarding managing multiple versions with your operator, I'm not sure if there are established patterns but you could either have several reconcilers that each deal with different versions of your API or you could deploy different versions of your operator per tenant depending on the API/CRD version they require.

[stefanbethke]

I've successfully implemented using a ConfigMap as an alternative, or in addition to, the CRD. It turned out to be relatively straight forward, I just needed to rerfactor the CRD bean to disentangle the spec and the status from the actual resource, and move the reconcile method in to my target state abstraction. Have a look here if you're interested: https://github.com/T-Systems-MMS/cmcc-operator

[csviri]

thank you! happy to hear that! will take a look.