EventHubs Scaler With Dapr - Won't Scale Beyond Single Pod

[peter-alegeus]

Good afternoon, I am creating a POC that uses .Net 6, Dapr, and Keda running on a local minikube, deployed via helm chart. The Daprized app is a standard web API used as a pubsub subscriber to eventhubs. The overall app works fine; I can send messages to the subscriber and it can process them, with a 5s delay. I recently added Keda to it and have stumbled a bit; I can't seem to get the deployment for the pubsub subscriber (named dapr-pubsub-keda) to spawn more than a single pod instance. Note that it also does not scale down to 0, other than a single time. Yesterday I applied an update to minikube, the kubernetes version with, and rebooted. When minikube came up my pod had scaled down to 0! I added messages to the queue and it quickly spawned a single pod. I currently have the unprocessedEventThreshold set to 5 and I send anywhere from 50 to 1000 messages, so it should scale past a single pod. I tried adding a failure condition to the spec to go to 3 pods just in case Keda was unsuccessfully pulling info from EventHubs, but since I'm still at a single pod and nothing shows in the log about using the failure condition, I don't think this helped any. I'll happily provide any information that can be of use. For starters, here's the yaml for the ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: eventhubs-scaledobject-keda
  namespace: default
spec:
  maxReplicaCount: 5
  pollingInterval: 5  # Optional. Default: 30 seconds
  cooldownPeriod:  10 # Optional. Default: 300 seconds
  fallback:
    failureThreshold: 3
    replicas: 3
  scaleTargetRef:
    name: dapr-pubsub-keda
    #envSourceContainerName: dapr-pubsub-poc
  triggers:
    - type: azure-eventhub
      metadata:
        connectionFromEnv: EVENTHUB_CONNECTION
        storageConnectionFromEnv: EVENTHUB_STORAGE_CONNECTION
        eventHubNamespace: alegeus-peter-dapr
        eventHubName: alegeus-peter-dapr-keda
        consumerGroup: dapr-pubsub-keda
        unprocessedEventThreshold: '5'
        blobContainer: peterdaprqueuecontainerkeda
        checkpointStrategy: blobMetadata
---

I'm running on Keda v2.4, minikube v1.24.0, k8s version 1.22.3 (client is 1.22.0), and using Dapr 1.3.0.

I've checked the logs of the keda operator and the keda metrics operator, and while I'm new to this nothing in there looks fishy to me. Will happily provide the info within if requested.

Any help would be appreciated, thanks.

[peter-alegeus]

Adding to the information given above, I pulled the helm chart down local and enabled Debug level logging. The metrics operator definitely recognizes some event hub messages. This first line shows that directly after reinstall, it thinks there are 13000 events within the hub:

I1116 22:05:36.593496 1 azure_eventhub_scaler.go:276] keda_metrics_adapter/azure_eventhub_scaler "msg"="Partition ID: 0, Last Enqueued Offset: 8901936, Checkpoint Offset: , Total new events in partition: 13000"
I1116 22:05:36.593529 1 azure_eventhub_scaler.go:283] keda_metrics_adapter/azure_eventhub_scaler "msg"="Unprocessed events in event hub total: 13000, scaling for a lag of 5 related to 1 partitions"

I then created 500 more events in the same hub+consumer group and got this:

I1116 22:06:36.815749 1 azure_eventhub_scaler.go:276] keda_metrics_adapter/azure_eventhub_scaler "msg"="Partition ID: 0, Last Enqueued Offset: 9169936, Checkpoint Offset: , Total new events in partition: 13500"
I1116 22:06:36.815774 1 azure_eventhub_scaler.go:283] keda_metrics_adapter/azure_eventhub_scaler "msg"="Unprocessed events in event hub total: 13500, scaling for a lag of 5 related to 1 partitions"

I believe this shows that the metrics operator for Keda is finding the events, it's just not scaling properly based off of the findings. I'm still at a single pod of the relevant app despite having a maxReplicaCount of 5.

The HPA (the one auto-generated when the scaledobject is installed) shows this information:
Min replicas: 1
Max replicas: 5
Deployment pods: 1 current / 1 desired

[peter-alegeus]

A little more info: I tried changing the checkpointStrategy to goSdk, since the documentation says this is what Dapr uses. The same behavior is observed. I tried the default checkpoint strategy and there is no change. Also, regarding the above log, this line now stands out to me even more, due to the missing "Checkpoint Offset". I suspect this means that the checkpoint information from the Azure storage blob isn't parsing properly, and I further suspect it's because of the checkpointStrategy I've chosen vs what Dapr is using, but that should have been resolved by changing to goSdk and it did not change anything as far as I can tell.

I1116 22:06:36.815749 1 azure_eventhub_scaler.go:276] keda_metrics_adapter/azure_eventhub_scaler "msg"="Partition ID: 0, Last Enqueued Offset: 9169936, Checkpoint Offset: , Total new events in partition: 13500"

Does anyone know of a working Keda + EventHubs (optionally + Dapr) sample that works? I tried the one here on GitHub but it was built for Keda 1.0, and did not work properly with Keda 2.4. I had to modify a few things to get it to even install and run, but once I did it still did not scale properly. Note that another person has reported the Keda + EventHubs sample doesn't work, I don't believe he got a resolution.

[tomkerkhove]

@jeffhollan @ahmelsayed Can you help here?

Does anyone know of a working Keda + EventHubs (optionally + Dapr) sample that works? I tried the one here on GitHub but it was built for Keda 1.0, and did not work properly with Keda 2.4.

Can you elaborate on this please? Are you referring to the sample?

[peter-alegeus]

Yes I am referring to the sample here. There are discussion threads that mention the sample is not updated to work with Keda 2+. I briefly tried to update it myself to make it work in my local minikube instance while trying to debug my own project but abandoned the effort and went another direction.