Pods spawned by KEDA are immediately terminated even whilst their scaling threshold is crossed #5350

tonetechnician · 2024-01-05T12:30:21Z

tonetechnician
Jan 5, 2024

Hi there,

I've setup KEDA to consume from a kafka message queue. The deployment seems to work correctly and the HPA is set up. When I I send messages to the kafka queue, KEDA scales the pods up correctly.

Unfortunately though, as soon as KEDA starts up the pods, the pods are then almost immediately terminated even while the threshold is still crossed. I'm unsure why this would be happening. I've attached the ScaledObject manifest here for reference

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: test-scaled-object
  namespace: backend
spec:
  scaleTargetRef:
    apiVersion:    apps/v1
    kind:          Deployment
    name:          test
  pollingInterval:  5
  cooldownPeriod:   1200
  maxReplicaCount:  3
  triggers:
  - type: kafka
    metricType: AverageValue
    metadata:
      bootstrapServers: <to be patched by infrustructure overlay>
      consumerGroup: <to be patched by infrustructure overlay>
      topic: <to be patched by infrustructure overlay>
      lagThreshold: '5'
      activationLagThreshold: '1'
      allowIdleConsumers: "true"
      offsetResetPolicy: latest
      scaleToZeroOnInvalidOffset: 'false'
      excludePersistentLag: 'false'
      version: 2.6.1

I'm unsure how to debug this correctly - are there logs anywhere where we can view what KEDA is doing and why these pods might be being terminated? I see that the HPA stays above the activation threshold even whilst the pods are being terminated.

I've attached the logs from the operator. We can see the total lag stays high, but during this time the pods that are created are immediately terminated after being created. Why?

2024-01-05T12:36:58Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:36:58Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:36:58Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:36:58Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:03Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:03Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:03Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:03Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:08Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:08Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:08Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:08Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:13Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:13Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:13Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:13Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:18Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:18Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:18Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:18Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:23Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:23Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:23Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:23Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:28Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:28Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:28Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:28Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:33Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:33Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:33Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:33Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:38Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 0, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:38Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:38Z","value":"0"}], "activity": false, "scalerError": null}
2024-01-05T12:37:38Z	DEBUG	scaleexecutor	ScaleTarget no change	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test"}
2024-01-05T12:37:43Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 7, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:43Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:43Z","value":"7"}], "activity": true, "scalerError": null}
2024-01-05T12:37:43Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:37:43Z	INFO	scaleexecutor	Successfully updated ScaleTarget	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-01-05T12:37:43Z	DEBUG	events	Scaled apps/v1.Deployment backend/test from 0 to 1	{"type": "Normal", "object": {"kind":"ScaledObject","namespace":"backend","name":"test-scaled-object","uid":"03b9cd87-c97e-4669-995a-b4973e5fbf0e","apiVersion":"keda.sh/v1alpha1","resourceVersion":"164016681"}, "reason": "KEDAScaleTargetActivated"}
2024-01-05T12:37:47Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 17, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:47Z	DEBUG	scale_handler	Getting metrics from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:47Z","value":"17"}], "scalerError": null}
2024-01-05T12:37:47Z	DEBUG	grpc_server	Providing metrics	{"scaledObjectName": "test-scaled-object", "scaledObjectNamespace": "backend", "metrics": "&ExternalMetricValueList{ListMeta:{   <nil>},Items:[]ExternalMetricValue{ExternalMetricValue{MetricName:s0-kafka-render_event_render_77dax88,MetricLabels:map[string]string{},Timestamp:2024-01-05 12:37:47.145940882 +0000 UTC m=+170658.602000740,WindowSeconds:nil,Value:{{17000 -3} {<nil>}  DecimalSI},},},}"}
2024-01-05T12:37:47Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "f06b7f24-3eb9-47f6-81ba-0de1a12b3c18"}
2024-01-05T12:37:47Z	DEBUG	Parsed Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "f06b7f24-3eb9-47f6-81ba-0de1a12b3c18", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2024-01-05T12:37:47Z	DEBUG	ScaledObject is defined correctly and is ready for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "f06b7f24-3eb9-47f6-81ba-0de1a12b3c18"}
2024-01-05T12:37:48Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 17, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:48Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:48Z","value":"17"}], "activity": true, "scalerError": null}
2024-01-05T12:37:48Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:37:53Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 33, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:53Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:53Z","value":"33"}], "activity": true, "scalerError": null}
2024-01-05T12:37:53Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:37:58Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 33, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:37:58Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:37:58Z","value":"33"}], "activity": true, "scalerError": null}
2024-01-05T12:37:58Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:38:02Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "32f219a1-392f-4d97-9825-a2227dd3a0ad"}
2024-01-05T12:38:02Z	DEBUG	Parsed Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "32f219a1-392f-4d97-9825-a2227dd3a0ad", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2024-01-05T12:38:02Z	DEBUG	ScaledObject is defined correctly and is ready for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "32f219a1-392f-4d97-9825-a2227dd3a0ad"}
2024-01-05T12:38:03Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 50, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:38:03Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:38:03Z","value":"50"}], "activity": true, "scalerError": null}
2024-01-05T12:38:03Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:38:03Z	INFO	scaleexecutor	Successfully updated ScaleTarget	{"scaledobject.Name": "test-scaled-object", "scaledObject.Namespace": "backend", "scaleTarget.Name": "test", "Original Replicas Count": 0, "New Replicas Count": 1}
2024-01-05T12:38:03Z	DEBUG	events	Scaled apps/v1.Deployment backend/test from 0 to 1	{"type": "Normal", "object": {"kind":"ScaledObject","namespace":"backend","name":"test-scaled-object","uid":"03b9cd87-c97e-4669-995a-b4973e5fbf0e","apiVersion":"keda.sh/v1alpha1","resourceVersion":"164017959"}, "reason": "KEDAScaleTargetActivated"}
2024-01-05T12:38:08Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 50, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:38:08Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:38:08Z","value":"50"}], "activity": true, "scalerError": null}
2024-01-05T12:38:08Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:38:13Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 50, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:38:13Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:38:13Z","value":"50"}], "activity": true, "scalerError": null}
2024-01-05T12:38:13Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:38:17Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 50, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:38:17Z	DEBUG	scale_handler	Getting metrics from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:38:17Z","value":"50"}], "scalerError": null}
2024-01-05T12:38:17Z	DEBUG	grpc_server	Providing metrics	{"scaledObjectName": "test-scaled-object", "scaledObjectNamespace": "backend", "metrics": "&ExternalMetricValueList{ListMeta:{   <nil>},Items:[]ExternalMetricValue{ExternalMetricValue{MetricName:s0-kafka-render_event_render_77dax88,MetricLabels:map[string]string{},Timestamp:2024-01-05 12:38:17.279509942 +0000 UTC m=+170688.735569797,WindowSeconds:nil,Value:{{50000 -3} {<nil>}  DecimalSI},},},}"}
2024-01-05T12:38:17Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "4f731366-66d7-473b-a5ba-a38a7c6506e5"}
2024-01-05T12:38:17Z	DEBUG	Parsed Group, Version, Kind, Resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "4f731366-66d7-473b-a5ba-a38a7c6506e5", "GVK": "apps/v1.Deployment", "Resource": "deployments"}
2024-01-05T12:38:17Z	DEBUG	ScaledObject is defined correctly and is ready for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"test-scaled-object","namespace":"backend"}, "namespace": "backend", "name": "test-scaled-object", "reconcileID": "4f731366-66d7-473b-a5ba-a38a7c6506e5"}
2024-01-05T12:38:18Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 50, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:38:18Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:38:18Z","value":"50"}], "activity": true, "scalerError": null}
2024-01-05T12:38:18Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}
2024-01-05T12:38:23Z	DEBUG	kafka_scaler	Kafka scaler: Providing metrics based on totalLag 50, topicPartitions 1, threshold 5	{"type": "ScaledObject", "namespace": "backend", "name": "test-scaled-object"}
2024-01-05T12:38:23Z	DEBUG	scale_handler	Getting metrics and activity from scaler	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88", "metrics": [{"metricName":"s0-kafka-render_event_render_77dax88","metricLabels":null,"timestamp":"2024-01-05T12:38:23Z","value":"50"}], "activity": true, "scalerError": null}
2024-01-05T12:38:23Z	DEBUG	scale_handler	Scaler for scaledObject is active	{"scaledObject.Namespace": "backend", "scaledObject.Name": "test-scaled-object", "scaler": "kafkaScaler", "metricName": "s0-kafka-render_event_render_77dax88"}

Answered by tonetechnician

Jan 5, 2024

Ah, so I figured the issue. In the Deployment manifest, you need to omit the spec.replicas field otherwise it tries to override the HPA replica settings. This issue helped #3578

View full answer

tonetechnician · 2024-01-05T13:59:24Z

tonetechnician
Jan 5, 2024
Author

Ah, so I figured the issue. In the Deployment manifest, you need to omit the spec.replicas field otherwise it tries to override the HPA replica settings. This issue helped #3578

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pods spawned by KEDA are immediately terminated even whilst their scaling threshold is crossed #5350

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

Pods spawned by KEDA are immediately terminated even whilst their scaling threshold is crossed #5350

tonetechnician Jan 5, 2024

Replies: 1 comment

tonetechnician Jan 5, 2024 Author

tonetechnician
Jan 5, 2024

tonetechnician
Jan 5, 2024
Author