[GH-ISSUE #395] [FEATURE REQUEST] Distributed Scheduler #174

New issue

Closed

opened 2026-03-02 05:19:19 +03:00 by kerem · 3 comments

kerem commented

2026-03-02 05:19:19 +03:00

Owner

Originally created by @aakashbajaj on GitHub (Feb 3, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/395

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
RIght now, a dedicated scheduler needs to be running. If that instance/service goes down, there's no way of recovery (specifically in kubernetes pod)

Describe the solution you'd like
Multiple schedulers register themselves on redis and maintain a master copy of Scheduler Entries. So, that if one instance goes down, other can pick right back up from there.

Originally created by @aakashbajaj on GitHub (Feb 3, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/395 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** RIght now, a dedicated scheduler needs to be running. If that instance/service goes down, there's no way of recovery (specifically in kubernetes pod) **Describe the solution you'd like** Multiple schedulers register themselves on redis and maintain a master copy of Scheduler Entries. So, that if one instance goes down, other can pick right back up from there.

kerem

2026-03-02 05:19:19 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-02 05:19:20 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 4, 2022):

@aakashbajaj Thank you for opening an issue!

Can you run a multiple instances of PeriodicTaskManager (see wiki here for more details) and store the periodic task configuration in a durable storage (e.g. Database, file in a distrubted filesystem, etc) and source that with PeriodicTaskConfigProvider?

To prevent these multiple instances of PeriodicTaskManagers from enqueueing duplicate tasks, you should be able to use the combinations of the following task options.

TaskID(uniqueID)
Retention(duration)
(This is kind of a workaround since the current Unique option doesn't work well if the task gets processed quickly, see the discussion below for details).

This was previously discussed here in this discussion.

Let me know if you have more questions or feedback!

@hibiken commented on GitHub (Feb 4, 2022): @aakashbajaj Thank you for opening an issue! Can you run a multiple instances of `PeriodicTaskManager` ([see wiki here for more details](https://github.com/hibiken/asynq/wiki/Dynamic-Periodic-Task)) and store the periodic task configuration in a durable storage (e.g. Database, file in a distrubted filesystem, etc) and source that with `PeriodicTaskConfigProvider`? To prevent these multiple instances of `PeriodicTaskManager`s from enqueueing duplicate tasks, you should be able to use the combinations of the following task options. - `TaskID(uniqueID)` - `Retention(duration)` (This is kind of a workaround since the current `Unique` option doesn't work well if the task gets processed quickly, see the discussion below for details). This was previously discussed here in [this discussion](https://github.com/hibiken/asynq/discussions/376). Let me know if you have more questions or feedback!

kerem commented

2026-03-02 05:19:20 +03:00

Author

Owner

@aakashbajaj commented on GitHub (Feb 7, 2022):

Thanks @hibiken for the suggestion. I'll look into this.

@aakashbajaj commented on GitHub (Feb 7, 2022): Thanks @hibiken for the suggestion. I'll look into this.

kerem commented

2026-03-02 05:19:21 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 8, 2022):

@aakashbajaj Closing this for now, but feel free to open this with a follow-up question or feedback 👍

@hibiken commented on GitHub (Feb 8, 2022): @aakashbajaj Closing this for now, but feel free to open this with a follow-up question or feedback 👍

kerem referenced this issue

2026-03-02 05:54:25 +03:00

[PR #174] [MERGED] Automatic recovery of orphan tasks #650

kerem referenced this issue

2026-03-07 22:11:47 +03:00

[PR #174] [MERGED] Automatic recovery of orphan tasks #1657

kerem referenced this issue

2026-03-15 20:56:54 +03:00

[PR #174] [MERGED] Automatic recovery of orphan tasks #2670