[GH-ISSUE #308] [FEATURE REQUEST] Scheduler reboot cannot reschedule old tasks

kerem commented

2026-03-07 22:06:24 +03:00

Owner

Originally created by @bibilins on GitHub (Aug 5, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/308

Originally assigned to: @hibiken on GitHub.

I want to know, if the server/scheduler roboot, will the tasks that were previously enqueued but not executed be rescheduled？

Originally created by @bibilins on GitHub (Aug 5, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/308 Originally assigned to: @hibiken on GitHub. I want to know, if the server/scheduler roboot, will the tasks that were previously enqueued but not executed be rescheduled？

kerem

2026-03-07 22:06:24 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@bibilins commented on GitHub (Aug 5, 2021):

I had tested, enqueued a task into scheduler, then reboot the scheduler, but it seems that scheduler was not scheduled the previous task

@bibilins commented on GitHub (Aug 5, 2021): I had tested, enqueued a task into scheduler, then reboot the scheduler, but it seems that scheduler was not scheduled the previous task

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@JeremyCraven commented on GitHub (Aug 5, 2021):

I am also curious about persistence of registered tasks with the scheduler.

My use case is scheduling a task to run on a schedule for each user only after the user performs a specific action for the first time. Currently as far as I'm aware, the only way to support this use case is determining all the users eligible for registering tasks and registering them with the scheduler before starting the scheduler. Would appreciate any guidance.

@JeremyCraven commented on GitHub (Aug 5, 2021): I am also curious about persistence of registered tasks with the scheduler. My use case is scheduling a task to run on a schedule for each user only after the user performs a specific action for the first time. Currently as far as I'm aware, the only way to support this use case is determining all the users eligible for registering tasks and registering them with the scheduler before starting the scheduler. Would appreciate any guidance.

kerem commented

2026-03-07 22:06:25 +03:00

Author

Owner

@crossworth commented on GitHub (Aug 5, 2021):

Hello guys, I will try to explain how the scheduler works, but I may be wrong on some aspects, so maybe we should wait to see what @hibiken says.

Lets think about asynq as two parts:

Worker: the part of the code that you use to handle the task.
Client: the part of the code that you use to enqueue a task.

While the worker you can have only way to handle the task (registering a handler for it), the client you have two ways to enqueue a task.

You can manually call client.Enqueue or you can use the scheduler, when you use the scheduler, the task will not be put on the queue right away, it will be scheduled to run at the cronspec provided, than it will be put on the queue. So in theory you can have strange behaviour, like using the scheduler with a task with asynq.ProcessIn(24*time.Hour), that means that when the scheduled run it will enqueue a task to be process in 24 hours.

More in depth answer:

We use github.com/robfig/cron to handle the cron part, when we register a task using Register(cronspec string, task *Task, opts ...Option) (entryID string, err error), we create a new enqueueJob, that has a Run method to be called by robfig/cron according to the cronspec definition. The enqueueJob.Run method enqueues the task on the "normal" task handling workflow.

So if you register a cron task to be executed in 2 hours from now, starts the process and than stop the process in a few minutes, no task should be present on the task queue, since the process that enqueues the task has not been executed.

I hope that this explanation make sense.

I am also curious about persistence of registered tasks with the scheduler.

My use case is scheduling a task to run on a schedule for each user only after the user performs a specific action for the first time. Currently as far as I'm aware, the only way to support this use case is determining all the users eligible for registering tasks and registering them with the scheduler before starting the scheduler. Would appreciate any guidance.

I dont think you need to register the task before starting the scheduler, the Register method internally calls AddJob and the documentation says that you can add jobs after the scheduler has been started, but there is a map write without a lock, so it may panic with concurrent write panic. Make sure you have the task handler registered, since you cannot dynamic register task handlers.

On your use case, on scheduler startup you could loop all the users checking if they are allowed to perform the action and if so register the task, during runtime (new user register and the scheduler is already running) you could dynamic call register, I think this should work.

@crossworth commented on GitHub (Aug 5, 2021): Hello guys, I will try to explain how the scheduler works, but I may be wrong on some aspects, so maybe we should wait to see what @hibiken says. Lets think about `asynq` as two parts: - Worker: the part of the code that you use to handle the task. - Client: the part of the code that you use to enqueue a task. While the worker you can have only way to handle the task (registering a handler for it), the client you have two ways to enqueue a task. You can manually call `client.Enqueue` or you can use the scheduler, when you use the scheduler, the task will not be put on the queue right away, it will be scheduled to run at the cronspec provided, than it will be put on the queue. So in theory you can have strange behaviour, like using the scheduler with a task with `asynq.ProcessIn(24*time.Hour)`, that means that when the scheduled run it will enqueue a task to be process in 24 hours. More in depth answer: We use [`github.com/robfig/cron` to handle the cron part](https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/scheduler.go#L28), when we register a task using `Register(cronspec string, task *Task, opts ...Option) (entryID string, err error)`, we create a new [enqueueJob](https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/scheduler.go#L107), that has a `Run` method to be called by `robfig/cron` according to the cronspec definition. The [`enqueueJob.Run`](https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/scheduler.go#L119) method enqueues the task on the "normal" task handling workflow. So if you register a cron task to be executed in 2 hours from now, starts the process and than stop the process in a few minutes, no task should be present on the task queue, since the process that enqueues the task has not been executed. I hope that this explanation make sense. > > > I am also curious about persistence of registered tasks with the scheduler. > > My use case is scheduling a task to run on a schedule for each user only after the user performs a specific action for the first time. Currently as far as I'm aware, the only way to support this use case is determining all the users eligible for registering tasks and registering them with the scheduler before starting the scheduler. Would appreciate any guidance. I dont think you need to register the task before starting the scheduler, the `Register` method internally calls `AddJob` and the documentation says that [you can add jobs after the scheduler has been started](https://pkg.go.dev/github.com/robfig/cron?utm_source=godoc#hdr-Usage), but there is a [map write](https://github.com/hibiken/asynq/blob/95c90a5cb822877a9465cd3203235a262359096f/scheduler.go#L157) without a lock, so it may panic with `concurrent write panic`. Make sure you have the task handler registered, since you cannot dynamic register task handlers. On your use case, on scheduler startup you could loop all the users checking if they are allowed to perform the action and if so register the task, during runtime (new user register and the scheduler is already running) you could dynamic call register, I think this should work.

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 6, 2021):

Thank you @bibilins for opening this issue!
@crossworth's explanation is correct. Current implementation is quite basic and it simply store all registered tasks and their cronspec in the process's memory (i.e doesn't persist it anyware), so it doesn't survive process restart.

What we want is a fault tolerant distributed cron system, I'll do some research and try to come up with a design and implementation.

Also, @JeremyCraven's use case is quite interesting. Maybe we should support dynamic registering of tasks after scheduler is launched. As @crossworth mentioned, we have to make sure to use mutex to guard that idmap. Feel free to open a PR if anyone's interested in contributing.

Thank you everyone for the feedback. This is a great opportunity to look into a more robust implementation of scheduler!

@hibiken commented on GitHub (Aug 6, 2021): Thank you @bibilins for opening this issue! @crossworth's explanation is correct. Current implementation is quite basic and it simply store all registered tasks and their cronspec in the process's memory (i.e doesn't persist it anyware), so it doesn't survive process restart. **What we want is a fault tolerant distributed cron system**, I'll do some research and try to come up with a design and implementation. Also, @JeremyCraven's use case is quite interesting. Maybe we should support dynamic registering of tasks after scheduler is launched. As @crossworth mentioned, we have to make sure to use mutex to guard that `idmap`. Feel free to open a PR if anyone's interested in contributing. Thank you everyone for the feedback. This is a great opportunity to look into a more robust implementation of scheduler!

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@JeremyCraven commented on GitHub (Aug 9, 2021):

Thanks @hibiken! We can work around registered tasks not surviving process restart.

I'll plan on following along and if the opportunity presents itself also help where possible. Thanks again!

@JeremyCraven commented on GitHub (Aug 9, 2021): Thanks @hibiken! We can work around registered tasks not surviving process restart. I'll plan on following along and if the opportunity presents itself also help where possible. Thanks again!

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@JeremyCraven commented on GitHub (Aug 16, 2021):

Thank you @bibilins for opening this issue!
@crossworth's explanation is correct. Current implementation is quite basic and it simply store all registered tasks and their cronspec in the process's memory (i.e doesn't persist it anyware), so it doesn't survive process restart.

What we want is a fault tolerant distributed cron system, I'll do some research and try to come up with a design and implementation.

Also, @JeremyCraven's use case is quite interesting. Maybe we should support dynamic registering of tasks after scheduler is launched. As @crossworth mentioned, we have to make sure to use mutex to guard that idmap. Feel free to open a PR if anyone's interested in contributing.

Thank you everyone for the feedback. This is a great opportunity to look into a more robust implementation of scheduler!

Hey @hibiken. Following up, it turned out that dynamic registering of tasks ended up being a blocker for us. I took a look at the scheduler code and it wasn't obvious how to implement that. If you have any suggestions, I could take an attempt at it, but either way just following up to express interest in this feature.

@JeremyCraven commented on GitHub (Aug 16, 2021): > Thank you @bibilins for opening this issue! > @crossworth's explanation is correct. Current implementation is quite basic and it simply store all registered tasks and their cronspec in the process's memory (i.e doesn't persist it anyware), so it doesn't survive process restart. > > **What we want is a fault tolerant distributed cron system**, I'll do some research and try to come up with a design and implementation. > > Also, @JeremyCraven's use case is quite interesting. Maybe we should support dynamic registering of tasks after scheduler is launched. As @crossworth mentioned, we have to make sure to use mutex to guard that `idmap`. Feel free to open a PR if anyone's interested in contributing. > > Thank you everyone for the feedback. This is a great opportunity to look into a more robust implementation of scheduler! Hey @hibiken. Following up, it turned out that dynamic registering of tasks ended up being a blocker for us. I took a look at the scheduler code and it wasn't obvious how to implement that. If you have any suggestions, I could take an attempt at it, but either way just following up to express interest in this feature.

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 16, 2021):

@JeremyCraven thank you for the update! I can revisit this and make the change within the next few days!

@hibiken commented on GitHub (Aug 16, 2021): @JeremyCraven thank you for the update! I can revisit this and make the change within the next few days!

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 18, 2021):

@JeremyCraven Please update the package to v0.18.4. I made a change so that you can call Scheduler.Register (and also Unregister) concurrently. Let me know if you encounter any issues!

@hibiken commented on GitHub (Aug 18, 2021): @JeremyCraven Please update the package to v0.18.4. I made a change so that you can call `Scheduler.Register` (and also `Unregister`) concurrently. Let me know if you encounter any issues!

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@JeremyCraven commented on GitHub (Aug 19, 2021):

@hibiken Thank you so much! So far looks like we are unblocked.

@JeremyCraven commented on GitHub (Aug 19, 2021): @hibiken Thank you so much! So far looks like we are unblocked.

kerem commented

2026-03-07 22:06:26 +03:00

Author

Owner

@gf3 commented on GitHub (Oct 8, 2021):

@hibiken @crossworth i also have a use-case for a persisted/distributed schedule:

our platform allows shop owners to have multiple locations where goods are sold, each location has its own fulfillment time and rules. every order on our platform is associated with a location and has a fulfillment date in the future. therefore each location requires a daily job at a specific time to mark current orders as paid and then move tomorrow's orders into the "ready" state. of course we would like this schedule to persist across deploys and failures and we'd like to ensure the schedule works in a multi-node system, too.

@gf3 commented on GitHub (Oct 8, 2021): @hibiken @crossworth i also have a use-case for a persisted/distributed schedule: our platform allows shop owners to have multiple locations where goods are sold, each location has its own fulfillment time and rules. every order on our platform is associated with a location and has a fulfillment date in the future. therefore each location requires a daily job at a specific time to mark current orders as paid and then move tomorrow's orders into the "ready" state. of course we would like this schedule to persist across deploys and failures and we'd like to ensure the schedule works in a multi-node system, too. ![Store_Location_Order@2x](https://user-images.githubusercontent.com/18397/136492315-9e04a22d-93bc-4cc8-b66b-7549d1ebf0f4.png)

kerem commented

2026-03-07 22:06:27 +03:00

Author

Owner

@hibiken commented on GitHub (Jan 22, 2022):

Check out PeriodicTaskManager type, which was added in v0.21.
You can store your periodic task configuration in a durable storage (e.g. file, database, etc) and write an implementation of PeriodicTaskConfigProvider which exports that configuration. This also supports dynamically adding/removing periodic tasks.

See this wiki page for an example.

This is a new feature, so any feedback is appreciated!

@hibiken commented on GitHub (Jan 22, 2022): Check out `PeriodicTaskManager` type, which was added in v0.21. You can store your periodic task configuration in a durable storage (e.g. file, database, etc) and write an implementation of `PeriodicTaskConfigProvider` which exports that configuration. This also supports dynamically adding/removing periodic tasks. See [this wiki page](https://github.com/hibiken/asynq/wiki/Dynamic-Periodic-Task) for an example. This is a new feature, so any feedback is appreciated!

Rows
Columns

[GH-ISSUE #308] [FEATURE REQUEST] Scheduler reboot cannot reschedule old tasks #1142