Understanding the Functionality of the 'useCachedMetrics' Parameter

[dvergison]

I would like to understand how the parameter useCachedMetrics works.

Let's say I have 36 scalers, a pollingInterval of 30s, and a --horizontal-pod-autoscaler-sync-period of 15s.

How many calls will be made to Datadog? How frequently? In case of a metric change, what is the maximum amount of time that will elapse before the system takes it into account?

And how can I calculate all this myself ?

[zroubalik]

If you enable cached metrics, there would be a call from each scaler every 30s (pollingInterval), that's all. Based on that you can calculate the maxium time when the system will get updated metrics.

[tomkerkhove]

I believe this is also documented on https://keda.sh/docs/2.12/concepts/scaling-deployments/#caching-metrics, but I could be wrong

[soundofspace]

Hi, first of all thanks for the amazing work and responses here.

'useCachedMetrics' does indeed have some explanation available in the docs, but I find it rather vague and missing some crucial details that might be hard to understand without a deep dive in the code.
Some questions I have after after reading the docs and the related PRs:

• Docs mention this: Kubernetes (HPA controller) asks for a metric every few seconds then is this request routed to KEDA Metrics Server, that by default queries the scaler and reads the metric values , if I'm not mistaken this has now been changed to: hpa -> KEDA metrics server -> keda operator -> query scaler
• This feature enables caching of metric values during polling interval, how exactly does this caching logic work (polling=30s)?
  1. metrics server -> keda operator -> cache combined with, but running separately keda operator -> scaler (every 30s) (hpa has no external requests)
  2. metrics server -> keda operator -> cache <30s old return value, else -> scaler (for hpa with 15s, this means +- half requests are external)
  3. a hybrid solution of 1 and 2
• Is keda always polling metrics with PollingInterval or only when scaled to 0?

My current understanding after having a brief look at the code:

• keda always polls scalers with PollingInterval
• logic for caching is a hybrid, but is mainly using version 1, and is only using version 2 as a fallback when no cached metrics are found. Hpa's would only very rarely make external requests.

Main reason for asking this, is because we are having problems with hpa's scaling slow because of high latency to keda metrics server because of slow external metrics. We are hoping caching metrics will (at least partially) solve this problem, and are trying to calculate/understand the impact of this change.

The exact implementation of this caching logic actually has a pretty big impact on the end result, so I'm hoping to shed some more light on this with these questions.

[zroubalik]

Hi, yes your assumption is correct.

• KEDA always polls scalers during polling interval and this is used for activation (ie scaling 0<->1). When useCachedMetrics: true, then the value is stored.
• HPA controller queries metrics every few seconds (usually 15s), if useCachedMetrics: false this results in the query to the external service. If useCachedMetrics: true then the value from cache (populated during polling interval) is used, if not found in the cache, then it queries the external service.