Scale KEDA Scaled Job

[aeolus3000]

Hello Community,

our team is using KEDA ScaledJobs and we are very happy with the scaling solution. It makes our lives much easier! Thank you for this great Open Source!

Now that we advance in our project we would like to create KEDA Scaled Jobs automatically to cater for different Resource Demands of our k8s Jobs.
As of now, we are deploying 3 KEDA Scaled Jobs at Deployment time.
In future we would like to add KEDA Scaled Jobs dynamically.
In the image below you can see that our current concept for the future architecture foresees a new component that monitors a redis list. The content of the redis set is maintained dynamically based on the tasks that need to be performed by the cluster. Based on the redis list another Custom Controller will create KEDA Scaled Jobs that are in turn observed by the KEDA Custom Controller.

This looks pretty straight forward and is probably the way other Controllers in k8s work.

I also understood that the KEDA Custom Controller will primarily create exact copies of k8s Jobs based on the template provided in the KEDA Scaled Job specification.

Here is the question:
I see an opportunity of integrating this behaviour into KEDA based on the vision below. Is this something that the community would support or need too?

Vision:
I could imagine to store all properties required to define a KEDA Scaled Job (minReplicaCount, maxReplicaCount, values to be replaced in the k8s Job) in a Redis List. This means that a Redis List exists that contain additional KEDA Scaled Jobs. Then, not only the actual KEDA Scaled Jobs applied to the cluster will be observed by the KEDA Custom Controller but also those that are listed in Redis.

Of course redis may only be one of the datasources.

[zroubalik]

@aeolus3000 thanks for bringing up this topic. I somehow missed this. This is indeed interesting, though I would prefer if we have a more generic solution (not something baked in Redis). What do you think?

[aeolus3000]

Hi @zroubalik, thank you for your reply. I absolutely agree. Redis should only be one of many possible data sources if this becomes open source. I tried to point this out in the very last sentence: "Of course redis may only be one of the datasources."

[aeolus3000]

How to move on in the open source world?