DatabaseScheduler is bad for a large amount of tasks?

[farahats9]

Summary:

Hello, I am using celery beat to work with a large amount of future tasks in the database, It is starting to struggle when the number of tasks enabled is 100k+ (there are hundreds of thousands of disabled tasks)

The tasks do execute but after their scheduled time by a significant amount (few hours) which is not good for my use case, I tried scaling up the server it helped a little but the issue still exists and the number of tasks is going to increase over time.
Also increasing the number of workers doesn't seem to affect the issue very much.

Am I doing something wrong or is this the limit of the tool? any alternative solutions are very appreciated.

• Celery Version: 5.3.6
• Celery-Beat Version: 2.5.0

Exact steps to reproduce the issue:

1. Create thousands of tasks with different schedules.
2. Wait for the tasks to execute on time
3. Notice they are delayed from the scheduled time.