Multiple SWS ScaledObjects not working

[yapjinkun]

Hi,

I'm trying to setup multiple sqs endpoints, each sqs endpoint will have it's own deployment target under same EKS cluster.

However, KEDA only works when I only have 1 scaledobject, if I created more than 1, all scaledObject will stopped working until I deleted and remain only 1 scaledobject in my cluster.

Below are my scaledobject yaml files

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: aws-sqs-queue-scaledobject
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-app-1 
  minReplicaCount: 1  
  maxReplicaCount: 100
  pollingInterval: 30
  cooldownPeriod:  10
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: keda-aws-credentials
    metadata:
      queueURL: https://sqs.ap-southeast-1.amazonaws.com/123456/KEDA-test.fifo
      queueLength: "2"
      awsRegion: ap-southeast-1
      identityOwner: operator

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: aws-sqs-queue-scaledobject-alt
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: deployment-app-2
  minReplicaCount: 1  
  maxReplicaCount: 100
  pollingInterval: 30 
  cooldownPeriod:  10 
  triggers:
  - type: aws-sqs-queue
    authenticationRef:
      name: keda-aws-credentials
    metadata:
      queueURL: https://sqs.ap-southeast-1.amazonaws.com/123456/KEDA-test2.fifo
      queueLength: "3"
      awsRegion: ap-southeast-1
      identityOwner: operator

Is this a limitation/bug/something i missed out.

[zroubalik]

This should work, probably a misconfiguration?

What is in the KEDA operator logs?

[yapjinkun]

thanks for reply!.

from my understanding that should works but everything stopped working until i deleted one of the scaledobject (either one), then the other scaledobject that remain will start working immediately (either scale up / down the targeted deployment).

this is the log from operator.

2024-06-06T08:45:50Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4524b24c-852c-45a4-a738-7ca827268e82"}
2024-06-06T08:45:50Z	INFO	Adding Finalizer for the ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4524b24c-852c-45a4-a738-7ca827268e82"}
2024-06-06T08:45:50Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4524b24c-852c-45a4-a738-7ca827268e82", "resource": "apps/v1.Deployment", "name": "inflate2"}
2024-06-06T08:45:50Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4524b24c-852c-45a4-a738-7ca827268e82", "HPA.Namespace": "default", "HPA.Name": "keda-hpa-aws-sqs-queue-scaledobject-alt"}
2024-06-06T08:45:50Z	INFO	scale_handler	WARNING: AWS EKS Identity has been deprecated (https://github.com/kedacore/keda/discussions/5343) and will be removed from KEDA on v3	{"type": "ScaledObject", "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt"}
2024-06-06T08:45:50Z	INFO	scale_handler	WARNING: AWS EKS Identity has been deprecated in favor of AWS Identity and will be removed from KEDA on v3	{"type": "ScaledObject", "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt"}
2024-06-06T08:45:50Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4524b24c-852c-45a4-a738-7ca827268e82"}
2024-06-06T08:45:50Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "6e391471-9696-444f-bb5d-0d1da3098ed6"}
2024-06-06T08:45:50Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "6e391471-9696-444f-bb5d-0d1da3098ed6", "resource": "apps/v1.Deployment", "name": "inflate2"}
2024-06-06T11:37:51Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4fc8ddcb-57cf-408c-8bdd-c777251c22eb"}
2024-06-06T11:37:51Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4fc8ddcb-57cf-408c-8bdd-c777251c22eb", "resource": "apps/v1.Deployment", "name": "inflate2"}

[yapjinkun]

after i deleted one of scaledobject, then it start to work again.

2024-06-06T11:37:51Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4fc8ddcb-57cf-408c-8bdd-c777251c22eb"}
2024-06-06T11:37:51Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject-alt","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject-alt", "reconcileID": "4fc8ddcb-57cf-408c-8bdd-c777251c22eb", "resource": "apps/v1.Deployment", "name": "inflate2"}
2024-06-06T15:00:32Z	INFO	cert-rotation	no cert refresh needed
2024-06-06T15:00:32Z	INFO	cert-rotation	Ensuring CA cert	{"name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "keda-admission", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-06-06T15:00:32Z	INFO	cert-rotation	Ensuring CA cert	{"name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService", "name": "v1beta1.external.metrics.k8s.io", "gvk": "apiregistration.k8s.io/v1, Kind=APIService"}
2024-06-06T15:47:19Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "7e01a3f1-3443-4791-8374-8719bd60452b"}
2024-06-06T15:47:19Z	INFO	Successfully finalized ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "7e01a3f1-3443-4791-8374-8719bd60452b"}

re-create scaledobject after deleted.

2024-06-06T15:49:33Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "042373b5-36dd-4bda-ad2f-98bf545fa64c"}
2024-06-06T15:49:33Z	INFO	Adding Finalizer for the ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "042373b5-36dd-4bda-ad2f-98bf545fa64c"}
2024-06-06T15:49:33Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "042373b5-36dd-4bda-ad2f-98bf545fa64c", "resource": "apps/v1.Deployment", "name": "inflate"}
2024-06-06T15:49:33Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "042373b5-36dd-4bda-ad2f-98bf545fa64c", "HPA.Namespace": "default", "HPA.Name": "keda-hpa-aws-sqs-queue-scaledobject"}
2024-06-06T15:49:33Z	INFO	scale_handler	WARNING: AWS EKS Identity has been deprecated (https://github.com/kedacore/keda/discussions/5343) and will be removed from KEDA on v3	{"type": "ScaledObject", "namespace": "default", "name": "aws-sqs-queue-scaledobject"}
2024-06-06T15:49:33Z	INFO	scale_handler	WARNING: AWS EKS Identity has been deprecated in favor of AWS Identity and will be removed from KEDA on v3	{"type": "ScaledObject", "namespace": "default", "name": "aws-sqs-queue-scaledobject"}
2024-06-06T15:49:33Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "042373b5-36dd-4bda-ad2f-98bf545fa64c"}
2024-06-06T15:49:33Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "a54e601b-5391-4093-9bf2-5cfd1012a598"}
2024-06-06T15:49:33Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"aws-sqs-queue-scaledobject","namespace":"default"}, "namespace": "default", "name": "aws-sqs-queue-scaledobject", "reconcileID": "a54e601b-5391-4093-9bf2-5cfd1012a598", "resource": "apps/v1.Deployment", "name": "inflate"}

no errors or anything, but it stopped working again.

[yapjinkun]

finally my KEDA setup is working for multi scaledobjects. I've been using awk-eks podidentity which have been deprecated but I never thought this is the one causing the problem since it's working fine for 1 scaledobject but no more than 1.

I changed it from "aws-eks" to "aws" and it's working fine for multi scaledobject now.

[zroubalik]

great, thanks for the update