Need help with deploying custom KEDA scaler

[sschimper]

I am working on developing a custom KEDA scaler for Splunk Observability Cloud, formerly known as SignalFX.
My environment consists of a single AWS EC2 instance, with Ubuntu and Microk8s installed.

I cloned the KEDA repository, created a custom scaler as described in CREATE-NEW-SCALER.md, built and deployed it according to the 'Deployment' instructions in BUILD.md.

To test the scaler, I am using a very simple Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

... and to activate the custom KEDA scaler the following Yaml file:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: nginx-scale-max
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx
  pollingInterval: 30
  cooldownPeriod: 300
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: splunk-o11y
    metricType: Value
    metadata:
      query: "data('demo.trans.latency').max().publish()" # SignalFx statement
      metricName: "demo.trans.latency"
      queryValue: "250.0" # Value to reach to start scaling
      activationQueryValue: "1.1"
      queryAggregator: "avg" # 'min', 'max', or 'avg'

In the logs of the operator I can see that the metrics are queried correctly. However, my deployment does not get scaled via the created HPA. When I run the 'describe' command on the HPA, I get:

Name:                                               keda-hpa-nginx-scale-max
Namespace:                                          default
Labels:                                             app.kubernetes.io/managed-by=keda-operator
                                                    app.kubernetes.io/name=keda-hpa-nginx-scale-max
                                                    app.kubernetes.io/part-of=nginx-scale-max
                                                    app.kubernetes.io/version=main
                                                    scaledobject.keda.sh/name=nginx-scale-max
Annotations:                                        <none>
CreationTimestamp:                                  Wed, 17 Jan 2024 08:19:16 +0000
Reference:                                          Deployment/nginx
Metrics:                                            ( current / target )
  "s0-signalfx-demo-trans-latency" (target value):  <unknown> / 250
Min replicas:                                       1
Max replicas:                                       10
Deployment pods:                                    0 current / 0 desired
Events:                                             <none>
root@i-052cbd7fcf277b0a0:/home/splunker# kubectl describe hpa keda-hpa-nginx-scale-max
Name:                                               keda-hpa-nginx-scale-max
Namespace:                                          default
Labels:                                             app.kubernetes.io/managed-by=keda-operator
                                                    app.kubernetes.io/name=keda-hpa-nginx-scale-max
                                                    app.kubernetes.io/part-of=nginx-scale-max
                                                    app.kubernetes.io/version=main
                                                    scaledobject.keda.sh/name=nginx-scale-max
Annotations:                                        <none>
CreationTimestamp:                                  Wed, 17 Jan 2024 08:19:16 +0000
Reference:                                          Deployment/nginx
Metrics:                                            ( current / target )
  "s0-signalfx-demo-trans-latency" (target value):  <unknown> / 250
Min replicas:                                       1
Max replicas:                                       10
Deployment pods:                                    1 current / 0 desired
Conditions:
  Type           Status  Reason                   Message
  ----           ------  ------                   -------
  AbleToScale    True    SucceededGetScale        the HPA controller was able to get the target's current scale
  ScalingActive  False   FailedGetExternalMetric  the HPA was unable to compute the replica count: unable to get external metric default/s0-signalfx-demo-trans-latency/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: nginx-scale-max,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server was unable to return a response in the time allotted, but may still be processing the request (get s0-signalfx-demo-trans-latency.external.metrics.k8s.io)
Events:
  Type     Reason                        Age                From                       Message
  ----     ------                        ----               ----                       -------
  Warning  FailedGetExternalMetric       18s (x2 over 78s)  horizontal-pod-autoscaler  unable to get external metric default/s0-signalfx-demo-trans-latency/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: nginx-scale-max,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server was unable to return a response in the time allotted, but may still be processing the request (get s0-signalfx-demo-trans-latency.external.metrics.k8s.io)
  Warning  FailedComputeMetricsReplicas  18s (x2 over 78s)  horizontal-pod-autoscaler  invalid metrics (1 invalid out of 1), first error is: failed to get s0-signalfx-demo-trans-latency external metric value: failed to get external metric s0-signalfx-demo-trans-latency: unable to get external metric default/s0-signalfx-demo-trans-latency/&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: nginx-scale-max,},MatchExpressions:[]LabelSelectorRequirement{},}: unable to fetch metrics from external metrics API: the server was unable to return a response in the time allotted, but may still be processing the request (get s0-signalfx-demo-trans-latency.external.metrics.k8s.io)

Here is an extract from the KEDA metrics API pod logs:

W0117 08:30:15.035680       1 logging.go:59] [core] [Channel #1 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "keda-operator.keda.svc.cluster.local:9666", ServerName: "keda-operator.keda.svc.cluster.local:9666", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.152.183.124:9666: connect: connection refused"
E0117 08:30:32.211336       1 provider.go:91] keda_metrics_adapter/provider "msg"="timeout" "error"="timeout while waiting to establish gRPC connection to KEDA Metrics Service server" "server"="keda-operator.keda.svc.cluster.local:9666"
I0117 08:30:32.211442       1 trace.go:236] Trace[772665059]: "List" accept:application/vnd.kubernetes.protobuf, */*,audit-id:54f39328-5fbd-42c8-8164-15f1b12fa323,client:127.0.0.1,protocol:HTTP/2.0,resource:s0-signalfx-demo-trans-latency,scope:namespace,url:/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s0-signalfx-demo-trans-latency,user-agent:kubelite/v1.29.0 (linux/amd64) kubernetes/3f7a50f/system:serviceaccount:kube-system:horizontal-pod-autoscaler,verb:LIST (17-Jan-2024 08:29:32.130) (total time: 60080ms):
Trace[772665059]: [1m0.080642511s] [1m0.080642511s] END
E0117 08:30:32.211651       1 timeout.go:142] post-timeout activity - time-elapsed: 86.835357ms, GET "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/s0-signalfx-demo-trans-latency" result: runtime error: invalid memory address or nil pointer dereference

For some reason, KEDA is unable to communicate with the Kubernetes HPA. So far, I failed to determine the cause of it. Could it have something to do with the way I set up my Microk8s cluster? The way I 'make run' the operator?
Any help is greatly appreciated. Thank you!

[zroubalik]

Sorry I totally missed this. Do I get it correctly, that you are running KEDA operator from a source code from your laptop and the rest (metrics server...) is in the K8s cluster? Then the metrics server is not able to resolve url to the operator. Easiest way is to deploy your changes in the actual image

[sschimper-splunk]

Hi Zbynek, no worries. I resolved the issue by deploying KEDA as an image as outlined in the "Custom KEDA as an image" section in the documentation.