Failing to setting up KEDA azure-queue ScaledObjects #2014

gc-sebastienle · 2021-08-06T10:14:24Z

gc-sebastienle
Aug 6, 2021

Greetings,

I am trying to set up KEDA with azure-queue so my deployment would scale-out if more than 2 messages are in queue.

I tried using both connectionFromEnv and authenticationRef but I'm failing both times with this error:

2021-08-06T09:20:26.188Z ERROR controllers.ScaledObject Failed to ensure HPA is correctly created for ScaledObject {"ScaledObject.Namespace": "my-demo-ns", "ScaledObject.Name": "my-azure-queue-3-rule", "error": "error getting scaler for trigger #0: error parsing azure queue metadata: no connection setting given"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-08-06T09:20:26.199Z ERROR controller Reconciler error {"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "my-azure-queue-3-rule", "namespace": "my-demo-ns", "error": "error getting scaler for trigger #0: error parsing azure queue metadata: no connection setting given"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

Here is my YAML file for using connectionFromEnv:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue-1
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-demo-ns-my-app
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 2   # Optional. Default: 0
  maxReplicaCount: 10  # Optional. Default: 100
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-1
      queueLength: '2'
      connectionFromEnv: AZUREQUEUESTORAGECS
      accountName: my-storage
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue-2
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-demo-ns-my-app
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 2   # Optional. Default: 0
  maxReplicaCount: 10  # Optional. Default: 100
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-2
      queueLength: '2'
      connectionFromEnv: AZUREQUEUESTORAGECS
      accountName: my-storage
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue-3
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-demo-ns-my-app
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 2   # Optional. Default: 0
  maxReplicaCount: 10  # Optional. Default: 100
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-3
      queueLength: '2'
      connectionFromEnv: AZUREQUEUESTORAGECS
      accountName: my-storage

AZUREQUEUESTORAGECS is properly set within deployment:

root@my-demo-ns-my-app-6b4bdb6z5-l4wl6:/# echo $AZUREQUEUESTORAGECS
DefaultEndpointsProtocol=https;AccountName=my-storage;AccountKey=V0aPXTlPDWg6dkId/fBSD73TO8lugdkwpfHIRbaPNFcUlQY1RSCqynNt1SQdDpQfquQUxwSFlkxL2XMUnSBc8Q==;EndpointSuffix=core.windows.net
root@my-demo-ns-my-app-6b4bdb6z5-l4wl6:/#

Here is my YAML file for using authenticationRef:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: my-demo-queue-auth
spec:
  podIdentity:
    provider: none
  secretTargetRef:
  - parameter: connection
    name: my-secret
    key: azure-storage-connectionstring
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue-1-rule
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-demo-ns-my-app
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 2   # Optional. Default: 0
  maxReplicaCount: 10  # Optional. Default: 100
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-1
      queueLength: '2'
      accountName: my-storage
    authenticationRef:
      name: my-demo-queue-auth
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue-2-rule
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-demo-ns-my-app
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 2   # Optional. Default: 0
  maxReplicaCount: 10  # Optional. Default: 100
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-2
      queueLength: '2'
      accountName: my-storage
      accountName: my-storage
    authenticationRef:
      name: my-demo-queue-auth
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue-3-rule
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-demo-ns-my-app
  pollingInterval: 10  # Optional. Default: 30 seconds
  cooldownPeriod:  300 # Optional. Default: 300 seconds
  minReplicaCount: 2   # Optional. Default: 0
  maxReplicaCount: 10  # Optional. Default: 100
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-3
      queueLength: '2'
      accountName: my-storage
    authenticationRef:
      name: my-demo-queue-auth

with the proper secret set as:

kubectl create secret generic my-secret --from-literal=azure-storage-connectionstring="DefaultEndpointsProtocol=https;AccountName=my-storage;AccountKey=V0aPXTlPDWg6dkId/fBSD73TO8lugdkwpfHIRbaPNFcUlQY1RSCqynNt1SQdDpQfquQUxwSFlkxL2XMUnSBc8Q==;EndpointSuffix=core.windows.net"

NOTE: I tried replacing the whole connectionString with its base64encode equivalent. There error remains the same.

kubectl create secret generic my-secret --from-literal=azure-storage-connectionstring="RGVmYXVsdEVuZHBvaW50c1Byb3RvY29sPWh0dHBzO0FjY291bnROYW1lPW15LXN0b3JhZ2U7QWNjb3VudEtleT1WMGFQWFRsUERXZzZka0lkL2ZCU0Q3M1RPOGx1Z2Rrd3BmSElSYmFQTkZjVWxRWTFSU0NxeW5OdDFTUWREcFFmcXVRVXh3U0Zsa3hMMlhNVW5TQmM4UT09O0VuZHBvaW50U3VmZml4PWNvcmUud2luZG93cy5uZXQ="

What am I missing?

Why KEDA is failing to connect to azure-queue storage?

Best Regards,

Answered by sebastienle14

Aug 10, 2021

I got it to work with the following confirguration:

first, I created a secret:

kubectl create secret generic my-secret --from-literal=azure-storage-connectionstring="DefaultEndpointsProtocol=https;AccountName=my-storage;AccountKey=V0aPXTlPDWg6dkId/fBSD73TO8lugdkwpfHIRbaPNFcUlQY1RSCqynNt1SQdDpQfquQUxwSFlkxL2XMUnSBc8Q==;EndpointSuffix=core.windows.net"

then I applied this configuration:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: my-queue-auth
spec:
  podIdentity:
    provider: none
  secretTargetRef:
  - parameter: connection
    name: my-secret
    key: azure-storage-connectionstring
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-az…

View full answer

tomkerkhove · 2021-08-06T11:39:14Z

tomkerkhove
Aug 6, 2021
Collaborator

The secret that you create is called batch-queue-cs-secret but you are referencing my-secret in the triggerauthentication, is that on purpose?

5 replies

zroubalik Aug 6, 2021
Maintainer

If you use connectionFromEnv the ENV needs to be set on the k8s Deployment (Container).

And I can see that you have 3 ScaledObjects pointing to the same Deployment, this won't work correctly, it will be interfering. Just define 3 triggers in one ScaledObject

zroubalik Aug 6, 2021
Maintainer

https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

gc-sebastienle Aug 6, 2021
Author

The secret that you create is called batch-queue-cs-secret but you are referencing my-secret in the triggerauthentication, is that on purpose?

sorry, this was a typo (I forgot to correct this part on publish)

meaning it doesn't work while the secret is named my-secret, sadly.

gc-sebastienle Aug 6, 2021
Author

If you use connectionFromEnv the ENV needs to be set on the k8s Deployment (Container).

And I can see that you have 3 ScaledObjects pointing to the same Deployment, this won't work correctly, it will be interfering. Just define 3 triggers in one ScaledObject

thank you, I will try this out ASAP and will post the results accordingly

sebastienle14 Aug 10, 2021

https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

I am still getting the error even though the env variable is now on the Kubernetes level:

2021-08-09T14:32:13.367Z	ERROR	controllers.ScaledObject	Failed to ensure HPA is correctly created for ScaledObject	{"ScaledObject.Namespace": "my-demo-ns", "ScaledObject.Name": "my-azure-queue", "error": "error getting scaler for trigger #0: error parsing azure queue metadata: no connection setting given"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:244
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
2021-08-09T14:32:13.385Z	ERROR	controller	Reconciler error	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "name": "my-azure-queue", "namespace": "my-demo-ns", "error": "error getting scaler for trigger #0: error parsing azure queue metadata: no connection setting given"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem

Here is the variable:

spec:
  containers:
  - name: my-app-container
    env:
    - name: AZUREQUEUESTORAGECS
      value: my-storage;AccountKey=V0aPXTlPDWg6dkId/fBSD73TO8lugdkwpfHIRbaPNFcUlQY1RSCqynNt1SQdDpQfquQUxwSFlkxL2XMUnSBc8Q==;EndpointSuffix=core.windows.net

However, I did solve this issue with the secret. Was it not working because I created the secret after the deployment? I do not know, but with secret created beforehand and deployment installed from scratch after. It did the trick. I will put another post with the YAML code and tag it as Answer

Thank you for your support!

sebastienle14 · 2021-08-10T12:50:22Z

sebastienle14
Aug 10, 2021

I got it to work with the following confirguration:

first, I created a secret:

kubectl create secret generic my-secret --from-literal=azure-storage-connectionstring="DefaultEndpointsProtocol=https;AccountName=my-storage;AccountKey=V0aPXTlPDWg6dkId/fBSD73TO8lugdkwpfHIRbaPNFcUlQY1RSCqynNt1SQdDpQfquQUxwSFlkxL2XMUnSBc8Q==;EndpointSuffix=core.windows.net"

then I applied this configuration:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: my-queue-auth
spec:
  podIdentity:
    provider: none
  secretTargetRef:
  - parameter: connection
    name: my-secret
    key: azure-storage-connectionstring
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: my-azure-queue
  namespace: my-demo-ns
spec:
  scaleTargetRef:
    name: my-app
  pollingInterval: 10
  cooldownPeriod:  300
  minReplicaCount: 2
  maxReplicaCount: 10
  advanced:
    restoreToOriginalReplicaCount: true
  triggers:
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-1
      queueLength: '2'
      accountName: my-storage
    authenticationRef:
      name: my-queue-auth
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-2
      queueLength: '2'
      accountName: my-storage
    authenticationRef:
      name: my-queue-auth
  - type: azure-queue
    metadata:
      queueName: my-azure-queue-3
      queueLength: '2'
      accountName: my-storage
    authenticationRef:
      name: my-queue-auth

It works perfectly!

Cheers to @zroubalik and @tomkerkhove for their time on this matter!

Best Regards,

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failing to setting up KEDA azure-queue ScaledObjects #2014

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Failing to setting up KEDA azure-queue ScaledObjects #2014

gc-sebastienle Aug 6, 2021

Replies: 2 comments · 5 replies

tomkerkhove Aug 6, 2021 Collaborator

zroubalik Aug 6, 2021 Maintainer

zroubalik Aug 6, 2021 Maintainer

gc-sebastienle Aug 6, 2021 Author

gc-sebastienle Aug 6, 2021 Author

sebastienle14 Aug 10, 2021

sebastienle14 Aug 10, 2021

gc-sebastienle
Aug 6, 2021

Replies: 2 comments 5 replies

tomkerkhove
Aug 6, 2021
Collaborator

zroubalik Aug 6, 2021
Maintainer

zroubalik Aug 6, 2021
Maintainer

gc-sebastienle Aug 6, 2021
Author

gc-sebastienle Aug 6, 2021
Author

sebastienle14
Aug 10, 2021