Backoff not respected due to resync

[sfiera]

I’ve been investigating an issue in our custom controller-manager where it appears that the backoff settings from service.Controller (exponential from 5s to 5m) weren’t respected. Our backend was timing out, and the controller retried soon after (within 30s), which compounded the issue.

I believe that this is due to the resync interval:

Down in processNextWorkItem(), at the end of a failed attempt, the controller will:

1. Schedule the service to be re-enqueued after backoff. (s.queue.AddRateLimited(key))
2. Remove the service from the queue. (s.queue.Done(key))
3. Wait for the backoff interval to elapse.
4. Re-enqueue the service.

However, if a resync happens, then service.Controller re-enqueues it without backoff. Because of the way that DelayingQueue works, an item is not (as far as I can tell) considered to be queued during (3), so the service key is not deduplicated as it would be for consecutive calls to Add().

The end result is that, despite maxRetryDelay of 5 minutes, service.Controller will never wait more than 30s before its next attempt to sync services.

There is a separate, related issue in kubernetes/client-go#131. It’s sort of the reverse. Here, I think that the call to AddRateLimited() should cancel a future call to Add(). There, the author wants a call to Add() to cancel a past call to AddRateLimited().

[sfiera]

I should add: resync removes loadbalancers, but does not add them. So, the backoff settings are ignored when deleting loadbalancers, but respected when adding or updating them.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[timoreimann]

/remove-lifecycle rotten

Thanks for reporting. This needs to be triaged.

[k8s-triage-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.