starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00

[GH-ISSUE #1013] [BUG]The issue of registering and destroying scheduled tasks when running multiple replicas. #2511

New issue

Open

opened 2026-03-15 20:44:51 +03:00 by kerem · 3 comments

kerem commented

2026-03-15 20:44:51 +03:00

Owner

Originally created by @guilinonline on GitHub (Jan 22, 2025).
Original GitHub issue: https://github.com/hibiken/asynq/issues/1013

Originally assigned to: @hibiken, @kamikazechaser on GitHub.

Describe the bug
When initializing an instance of asynq.NewScheduler, registering a scheduled task, and starting it, everything works fine. However, when the same code is deployed in a distributed manner (i.e., starting multiple instances), the same scheduled task is registered multiple times. This issue doesn't occur when only one instance is running.

Environment (please complete the following information):

OS: [e.g. Linux]
asynq package version: v0.25.1
Redis/Valkey version: 7.2.5
To Reproduce
Steps to reproduce the behavior:

Initialize a NewScheduler instance in your application.
Register a scheduled task.
Start multiple instances of the application in a distributed environment.
Observe that the same task is registered multiple times across instances.
Expected behavior
The scheduled task should only be registered once, regardless of the number of instances running in a distributed environment.

Screenshots
N/A

Additional context
This issue only arises when multiple instances of the application are running simultaneously. The task should be registered only once across all instances to avoid duplication.

two instance:

Originally created by @guilinonline on GitHub (Jan 22, 2025). Original GitHub issue: https://github.com/hibiken/asynq/issues/1013 Originally assigned to: @hibiken, @kamikazechaser on GitHub. Describe the bug When initializing an instance of asynq.NewScheduler, registering a scheduled task, and starting it, everything works fine. However, when the same code is deployed in a distributed manner (i.e., starting multiple instances), the same scheduled task is registered multiple times. This issue doesn't occur when only one instance is running. Environment (please complete the following information): OS: [e.g. Linux] asynq package version: v0.25.1 Redis/Valkey version: 7.2.5 To Reproduce Steps to reproduce the behavior: Initialize a NewScheduler instance in your application. Register a scheduled task. Start multiple instances of the application in a distributed environment. Observe that the same task is registered multiple times across instances. Expected behavior The scheduled task should only be registered once, regardless of the number of instances running in a distributed environment. Screenshots N/A Additional context This issue only arises when multiple instances of the application are running simultaneously. The task should be registered only once across all instances to avoid duplication. two instance: ![Image](https://github.com/user-attachments/assets/44d70e51-8a01-4952-adef-4b66d4e57334)

kerem added the

bug

label

2026-03-15 20:44:51 +03:00

kerem commented

2026-03-15 20:45:02 +03:00

Author

Owner

@guilinonline commented on GitHub (Jan 22, 2025):

I also believe that when multiple instances are running, the scheduled task should not be registered repeatedly, no matter which instance registers it. The scheduled task should only be stopped when the last instance shuts down. I think this approach aligns with the requirements of running multiple replicas.

@guilinonline commented on GitHub (Jan 22, 2025): I also believe that when multiple instances are running, the scheduled task should not be registered repeatedly, no matter which instance registers it. The scheduled task should only be stopped when the last instance shuts down. I think this approach aligns with the requirements of running multiple replicas.

kerem commented

2026-03-15 20:45:07 +03:00

Author

Owner

@guilinonline commented on GitHub (Jan 22, 2025):

I think I understand now. By adding a TaskID to the scheduler.Register parameters, when multiple instances are running, only one scheduled task will produce messages to the queue, and the load will be balanced across the instances. Is my understanding correct?

if _, err := scheduler.Register("* * * * *", asynq.NewTask(TaskTypeCalculateLeaderboard, []byte(`{"name":"xxxxxxx"}`)),
			asynq.Queue("test_cron"), asynq.TaskID("xxxxxdemo")); err != nil {
			log.Fatal(err)
		}

@guilinonline commented on GitHub (Jan 22, 2025): I think I understand now. By adding a TaskID to the scheduler.Register parameters, when multiple instances are running, only one scheduled task will produce messages to the queue, and the load will be balanced across the instances. Is my understanding correct? ``` if _, err := scheduler.Register("* * * * *", asynq.NewTask(TaskTypeCalculateLeaderboard, []byte(`{"name":"xxxxxxx"}`)), asynq.Queue("test_cron"), asynq.TaskID("xxxxxdemo")); err != nil { log.Fatal(err) } ```

kerem commented

2026-03-15 20:45:12 +03:00

Author

Owner

@gsaraf commented on GitHub (Mar 10, 2025):

Reading through the code - I think that might help a little bit, but it can still cause duplicates in a flow like:

Scheduler A: publishes task for instant K
Worker: completes task very quickly
Scheduler B: publishes task for same instant K <- there is no guarantee that the publishing for a certain instant will be synchronized across schedulers

It can also cause problems if the task wasn't completed in time before a second one is published:

Scheduler A: publishes task for instant K
Worker: busy, does not process
Scheduler A: publishes task for instant K+1 <- Not actually enqueued since ID exists already.

Perhaps the correct solution would be to have only one worker publish tasks to the scheduler? You can acquire a lock (over redis also) to synchronize between them.

@gsaraf commented on GitHub (Mar 10, 2025): Reading through the code - I think that might help a little bit, but it can still cause duplicates in a flow like: ``` Scheduler A: publishes task for instant K Worker: completes task very quickly Scheduler B: publishes task for same instant K <- there is no guarantee that the publishing for a certain instant will be synchronized across schedulers ``` It can also cause problems if the task wasn't completed in time before a second one is published: ``` Scheduler A: publishes task for instant K Worker: busy, does not process Scheduler A: publishes task for instant K+1 <- Not actually enqueued since ID exists already. ``` Perhaps the correct solution would be to have only one worker publish tasks to the scheduler? You can acquire a lock (over redis also) to synchronize between them.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2511

No description provided.

Rows
Columns