fix(prometheus.operator): retry GetInformer when running prometheus operator components

[onematchfox]

PR Description

This PR adds a retry when attempting to run prometheus.operator components which ensures that component do not compltetely fail to start up if there is any form of delay in the pod being able to communicate with the Kubernetes API.

At present, the prometheus.operator flow components fail to run on our GKE clusters when a new pod starts up. Logs as follows:

{"ts":"2024-02-21T11:24:44.279996707Z","level":"error","msg":"error running crd manager","component":"prometheus.operator.podmonitors.pods","err":"could not create RESTMapper from config: Get \"https://<redacted>:443/api\": dial tcp <redacted>:443: connect: connection refused"}
{"ts":"2024-02-21T11:24:44.280125067Z","level":"error","msg":"error running crd manager","component":"prometheus.operator.servicemonitors.services","err":"could not create RESTMapper from config: Get \"https://<redacted>:443/api\": dial tcp <redacted>:443: connect: connection refused"}

However, if we force reload the config (remove the component and then add it back in again) then everything works as expected.

With these changes in place, the components successfully start after a retry or two. Logs as follows:

{"ts":"2024-02-21T13:39:44.328729137Z","level":"warn","msg":"failed to get informer - will retry","component":"prometheus.operator.servicemonitors.services","err":"failed to get restmapping: failed to get server groups: Get \"https://<redacted>:443/api\": dial tcp <redacted>:443: connect: connection refused"}
{"ts":"2024-02-21T13:39:44.328801047Z","level":"warn","msg":"failed to get informer - will retry","component":"prometheus.operator.podmonitors.pods","err":"failed to get restmapping: failed to get server groups: Get \"https://<redacted>:443/api\": dial tcp <redacted>:443: connect: connection refused"}
{"ts":"2024-02-21T13:39:44.479158994Z","level":"warn","msg":"failed to get informer - will retry","component":"prometheus.operator.podmonitors.pods","err":"failed to get restmapping: failed to get server groups: Get \"https://<redacted>:443/api\": dial tcp <redacted>:443: connect: connection refused"}
{"ts":"2024-02-21T13:39:44.521300633Z","level":"warn","msg":"failed to get informer - will retry","component":"prometheus.operator.servicemonitors.services","err":"failed to get restmapping: failed to get server groups: Get \"https://<redacted>:443/api\": dial tcp <redacted>:443: connect: connection refused"}
{"ts":"2024-02-21T13:39:44.977446245Z","level":"info","msg":"informers started","component":"prometheus.operator.podmonitors.pods"}
{"ts":"2024-02-21T13:39:45.008419174Z","level":"info","msg":"informers started","component":"prometheus.operator.servicemonitors.services"}

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

• CHANGELOG.md updated
• Documentation added
• Tests updated
• Config converters updated

[CLAassistant]

All committers have signed the CLA.

[onematchfox]

Change in mapper is required as the dynamic rest mapper lazy loads mappings. Hence, we can retry calls to GetInformer below rather than just failing when calling cache.New here. This is also the default going forward so rather than having to redo this work/risk introducing a regression (if I were to retry here instead), I have implemented this so that it will continue to work when upgrading controller-runtime.

[onematchfox]

See kubernetes-sigs/controller-runtime#2611 and https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.17.0. FWIW, I had a quick go (sorry, not sorry) at upgrading this but ran into also sorts of dependency issue due to the overrides in this repo (e.g. here).

[hainenber]

Is the comment meant to be in the go.mod? 👀

[onematchfox]

Yes, I intentionally put it here so that it will be visible in the diff when someone does upgrade this dependency

[hainenber]

Cool, thanks for clarifying!

[hainenber]

I think this assignment is a bit premature since if there are errors, opts attribute still keep the erroneous client

[onematchfox]

The code returns on error just below so opts will be discarded in that scenario regardless.

[hainenber]

Ah, good catch. LGTM then

[onematchfox]

Rebased to address conflicts introduced by #6552.

[github-actions]

This PR has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If you do not have enough time to follow up on this PR or you think it's no longer relevant, consider closing it.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your PR will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[github-actions]

This PR has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If you do not have enough time to follow up on this PR or you think it's no longer relevant, consider closing it.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your PR will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!

[onematchfox]

Waiting on maintainers

[github-actions]

This PR has not had any activity in the past 30 days, so the needs-attention label has been added to it.
If you do not have enough time to follow up on this PR or you think it's no longer relevant, consider closing it.
The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your PR will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity.
Thank you for your contributions!