Does KEDA support space aggregation (by) in DataDog query?

[nonylene]

DataDog metric queries can have a space aggregation (aka group by): https://docs.datadoghq.com/metrics/#anatomy-of-a-metric-query

Can we use the aggregation part in the query parameter of KEDA DataDog triggers?

Example query and use case

I am sending custom DataDog metrics that represent some utilization of each application instance (process). Each metric has a tag type and we want to scale Pods based on the maximum value within the each average utilization for each type.

For example, we can get the average utilization values for each type by using the following query:

avg:custom_utilization_metrics{app:example} by {type}

When any type exceeds a fixed threshold (queryAggregator: max), I want to scale out Pods.

Without space aggregator...

The query below returns the maximum value of the ALL metrics, which can be an outliner and can cause flapping.

max:custom_utilization_metrics{app:example}

Another query below returns the avg value of the ALL metrics. KEDA cannot scale out properly when only one type is highly utilized.

avg:custom_utilization_metrics{app:example}

As a workaround supported we can put queries for each (many!) types with multiple queries feature and set queryAggregator: max:

avg:custom_utilization_metrics{app:example, type:a}, avg:custom_utilization_metrics{app:example, type:b}, ... 

Current implementation status

I think KEDA can already handle multiple dataseries from space-aggregated queries, thanks to Multi-queries support in KEDA.

The response format from Query timeseries points API consumed by KEDA is common regardless of space aggregator existence or the number of queries; The API always returns multiple series of values in a series array.

For example, when we query 2 queries and the fist one has a space aggregator and the second one does not have, the response will be:

{
  ...,
  "series": [
    {series for the 1st query and the type A},
    {series for the 1st query and the type B},
    ..{series for the 1st query and the type ...}..,
    {series for the 2nd query}
  ]
}

We can handle all the series in the same way.

However, KEDA documentation does not mention space aggregators at all, so I am currently hesitant to use them in production. Can we use space aggregators in DataDog queries?