starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-26 07:25:56 +03:00

[GH-ISSUE #395] [FEATURE REQUEST] Distributed Scheduler #1189

New issue

Closed

opened 2026-03-07 22:06:57 +03:00 by kerem · 3 comments

kerem commented

2026-03-07 22:06:57 +03:00

Owner

Originally created by @aakashbajaj on GitHub (Feb 3, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/395

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
RIght now, a dedicated scheduler needs to be running. If that instance/service goes down, there's no way of recovery (specifically in kubernetes pod)

Describe the solution you'd like
Multiple schedulers register themselves on redis and maintain a master copy of Scheduler Entries. So, that if one instance goes down, other can pick right back up from there.

Originally created by @aakashbajaj on GitHub (Feb 3, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/395 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** RIght now, a dedicated scheduler needs to be running. If that instance/service goes down, there's no way of recovery (specifically in kubernetes pod) **Describe the solution you'd like** Multiple schedulers register themselves on redis and maintain a master copy of Scheduler Entries. So, that if one instance goes down, other can pick right back up from there.

kerem

2026-03-07 22:06:57 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 4, 2022):

@aakashbajaj Thank you for opening an issue!

Can you run a multiple instances of PeriodicTaskManager (see wiki here for more details) and store the periodic task configuration in a durable storage (e.g. Database, file in a distrubted filesystem, etc) and source that with PeriodicTaskConfigProvider?

To prevent these multiple instances of PeriodicTaskManagers from enqueueing duplicate tasks, you should be able to use the combinations of the following task options.

TaskID(uniqueID)
Retention(duration)
(This is kind of a workaround since the current Unique option doesn't work well if the task gets processed quickly, see the discussion below for details).

This was previously discussed here in this discussion.

Let me know if you have more questions or feedback!

@hibiken commented on GitHub (Feb 4, 2022): @aakashbajaj Thank you for opening an issue! Can you run a multiple instances of `PeriodicTaskManager` ([see wiki here for more details](https://github.com/hibiken/asynq/wiki/Dynamic-Periodic-Task)) and store the periodic task configuration in a durable storage (e.g. Database, file in a distrubted filesystem, etc) and source that with `PeriodicTaskConfigProvider`? To prevent these multiple instances of `PeriodicTaskManager`s from enqueueing duplicate tasks, you should be able to use the combinations of the following task options. - `TaskID(uniqueID)` - `Retention(duration)` (This is kind of a workaround since the current `Unique` option doesn't work well if the task gets processed quickly, see the discussion below for details). This was previously discussed here in [this discussion](https://github.com/hibiken/asynq/discussions/376). Let me know if you have more questions or feedback!

kerem commented

2026-03-07 22:06:59 +03:00

Author

Owner

@aakashbajaj commented on GitHub (Feb 7, 2022):

Thanks @hibiken for the suggestion. I'll look into this.

@aakashbajaj commented on GitHub (Feb 7, 2022): Thanks @hibiken for the suggestion. I'll look into this.

kerem commented

2026-03-07 22:07:00 +03:00

Author

Owner

@hibiken commented on GitHub (Feb 8, 2022):

@aakashbajaj Closing this for now, but feel free to open this with a follow-up question or feedback 👍

@hibiken commented on GitHub (Feb 8, 2022): @aakashbajaj Closing this for now, but feel free to open this with a follow-up question or feedback 👍

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1189

No description provided.

Rows
Columns