starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-26 15:35:55 +03:00

[GH-ISSUE #628] [QUESTION] is it possible to run different instances of asynq with same redis URL #2332

New issue

Open

opened 2026-03-15 20:08:57 +03:00 by kerem · 3 comments

kerem commented

2026-03-15 20:08:57 +03:00

Owner

Originally created by @TheDhejavu on GitHub (Mar 17, 2023).
Original GitHub issue: https://github.com/hibiken/asynq/issues/628

Originally assigned to: @hibiken on GitHub.

Question

i have two workers running different instances pointing to same redis URL , i did this but i noticed a weird clash between this instances.

For example:
A registered handler throws Last Failure: handler not found for task in one instance while the other instance work but lags.

Instance 1

// register job handlers
tsk := tasks.NewJobTask(cfg)
jobHandler := handler.NewJobHandler(service, cfg, tsk)
srv.AsynqSrvMux.HandleFunc(tasks.TypePageReCrawl, jobHandler.ProcessReCrawlPageRequest)

// Scheduler.....
if _, err := srv.AsynqScheduler.Register("@every 1m", tsk.NewPageRecrawlTask(tasks.TypePageReCrawl)); err != nil {
	log.Fatal().Err(err).Msg("unable to register scheduler")
}

Instance 2


// register job handlers
jobHandler := handler.NewJobHandler(service, cfg, tasks.NewJobTask(cfg))
srv.AsynqSrvMux.HandleFunc(tasks.TypePageCrawl, jobHandler.ProcessCrawlPageRequest)

I get this error in instance 2 with Last Failure: handler not found for task

What do you think?

Originally created by @TheDhejavu on GitHub (Mar 17, 2023). Original GitHub issue: https://github.com/hibiken/asynq/issues/628 Originally assigned to: @hibiken on GitHub. ## Question i have two workers running different instances pointing to same redis URL , i did this but i noticed a weird clash between this instances. For example: A registered handler throws `Last Failure: handler not found for task ` in one instance while the other instance work but lags. Instance 1 ```go // register job handlers tsk := tasks.NewJobTask(cfg) jobHandler := handler.NewJobHandler(service, cfg, tsk) srv.AsynqSrvMux.HandleFunc(tasks.TypePageReCrawl, jobHandler.ProcessReCrawlPageRequest) // Scheduler..... if _, err := srv.AsynqScheduler.Register("@every 1m", tsk.NewPageRecrawlTask(tasks.TypePageReCrawl)); err != nil { log.Fatal().Err(err).Msg("unable to register scheduler") } ``` Instance 2 ```go // register job handlers jobHandler := handler.NewJobHandler(service, cfg, tasks.NewJobTask(cfg)) srv.AsynqSrvMux.HandleFunc(tasks.TypePageCrawl, jobHandler.ProcessCrawlPageRequest) ``` I get this error in instance 2 with `Last Failure: handler not found for task ` <img width="660" alt="Screen Shot 2023-03-17 at 11 29 19 PM" src="https://user-images.githubusercontent.com/19678915/226063926-fe669c88-76cb-497b-9800-ed95ed0b5674.png"> What do you think?

kerem added the

enhancement

label

2026-03-15 20:08:57 +03:00

kerem commented

2026-03-15 20:09:08 +03:00

Author

Owner

@bfwbbrj commented on GitHub (Mar 28, 2023):

Did you specify Queues when you start up your service? Currently if you didn't do this, Default Queue Config will be used. And this will result in all instances fetching job from the same queue, thus causing your problem.

I personally believe this is better solved by adding the option to set prefix when starting new asynq instance, so that different instances can safely share the same redis instance. However, this will require some code changes. The alternative way to solve this is to set unique queue config for unique set of instance(s), so that tasks will be fetched by the correct asynq instance.

Note that if you choose the alternative solution, the inspector will show the collection of all queues.

@bfwbbrj commented on GitHub (Mar 28, 2023): Did you specify **[Queues](https://github.com/hibiken/asynq/blob/cc777ebdaa62b69bd6e985fa97117b854e7d1cd6/server.go#L140)** when you start up your service? Currently if you didn't do this, **[Default Queue Config](https://github.com/hibiken/asynq/blob/cc777ebdaa62b69bd6e985fa97117b854e7d1cd6/server.go#L378)** will be used. And this will result in all instances fetching job from the same queue, thus causing your problem. I personally believe this is **better solved** by adding the option to set prefix when starting new asynq instance, so that different instances can safely share the same redis instance. However, this will require some code changes. **The alternative way** to solve this is to **set unique queue config** for unique set of instance(s), so that tasks will be fetched by the correct asynq instance. Note that if you choose the alternative solution, the inspector will show the collection of all queues.

kerem commented

2026-03-15 20:09:13 +03:00

Author

Owner

@TheDhejavu commented on GitHub (Mar 29, 2023):

Hi @bfwbbrj , thank you , i was able to solve this using queues but i will rather like to have different instances with an option to set a prefix with different priority queues. This seems like a better solution to me.

@TheDhejavu commented on GitHub (Mar 29, 2023): Hi @bfwbbrj , thank you , i was able to solve this using queues but i will rather like to have different instances with an option to set a prefix with different priority queues. This seems like a better solution to me.

kerem commented

2026-03-15 20:09:18 +03:00

Author

Owner

@bfwbbrj commented on GitHub (Mar 30, 2023):

Hi @bfwbbrj , thank you , i was able to solve this using queues but i will rather like to have different instances with an option to set a prefix with different priority queues. This seems like a better solution to me.

I can open a pr for this once I have time. But since @hibiken has been busy recently, I'm not sure when will the pr be reviewed and merged.

@bfwbbrj commented on GitHub (Mar 30, 2023): > Hi @bfwbbrj , thank you , i was able to solve this using queues but i will rather like to have different instances with an option to set a prefix with different priority queues. This seems like a better solution to me. I can open a pr for this once I have time. But since @hibiken has been busy recently, I'm not sure when will the pr be reviewed and merged.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2332

No description provided.

Rows
Columns