[GH-ISSUE #314] [BUG] The high priority queue is process very slow compare to lower queue

kerem commented

2026-03-07 22:06:27 +03:00

Owner

Originally created by @duyhungtnn on GitHub (Aug 20, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/314

Originally assigned to: @hibiken on GitHub.

Describe the bug
The high priority queue is processed very slow compare to the lower queue

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):

Run a long-running task on the default queue. Many small but critical tasks will issue from here and added to the critical queue
The default queue is hold all the time of worker pools
The critical queue is stuck, and no task is process

Expected behavior

The critical queue's task need to be selected to process first and clean out quickly

Screenshots

Setup server with 3 queue
Enqueue critical task like that
The critical queue stuck
The default queue is running on all other workers

Environment (please complete the following information):

OS: Ubuntu 18.4
Version of asynq package 0.18.3

Originally created by @duyhungtnn on GitHub (Aug 20, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/314 Originally assigned to: @hibiken on GitHub. **Describe the bug** The high priority queue is processed very slow compare to the lower queue **To Reproduce** Steps to reproduce the behavior (Code snippets if applicable): - Run a long-running task on the `default` queue. Many small but critical tasks will issue from here and added to the `critical` queue - The `default` queue is hold all the time of worker pools - The `critical` queue is stuck, and no task is process **Expected behavior** - The `critical` queue's task need to be selected to process first and clean out quickly **Screenshots** - Setup server with 3 queue ![Screen Shot 2021-08-20 at 09 38 28](https://user-images.githubusercontent.com/2597710/130170972-db118f10-fb84-4c6d-b709-afaf249ac17c.png) - Enqueue critical task like that ![Screen Shot 2021-08-20 at 09 41 06](https://user-images.githubusercontent.com/2597710/130171039-76aa5d11-1941-40da-911a-c6a6bc0a427a.png) - The `critical` queue stuck ![Screen Shot 2021-08-20 at 09 37 49](https://user-images.githubusercontent.com/2597710/130171087-1c3727d5-991a-4bb7-8765-fab97549eb42.png) - The `default` queue is running on all other workers ![Screen Shot 2021-08-20 at 09 37 42](https://user-images.githubusercontent.com/2597710/130171114-04ef65a7-1fc5-4caa-9dd5-1ba744da98ff.png) **Environment (please complete the following information):** - OS: Ubuntu 18.4 - Version of `asynq` package `0.18.3` ![Screen Shot 2021-08-20 at 09 47 04](https://user-images.githubusercontent.com/2597710/130171244-2130f22a-bcb3-4e41-ba11-cc53b5cf92de.png)

kerem

2026-03-07 22:06:27 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-07 22:06:28 +03:00

Author

Owner

@crossworth commented on GitHub (Aug 20, 2021):

Hello @duyhungtnn, could you tell me what value are you passing in the Concurrency (ccu variable)?

@crossworth commented on GitHub (Aug 20, 2021): Hello @duyhungtnn, could you tell me what value are you passing in the Concurrency (ccu variable)?

kerem commented

2026-03-07 22:06:28 +03:00

Author

Owner

@duyhungtnn commented on GitHub (Aug 20, 2021):

Hi @crossworth , the Concurrency value is 6.

I have two asynq server (Concurrency value is 6 per server)

@duyhungtnn commented on GitHub (Aug 20, 2021): Hi @crossworth , the Concurrency value is 6. I have two asynq server (Concurrency value is 6 per server)

kerem commented

2026-03-07 22:06:28 +03:00

Author

Owner

@duyhungtnn commented on GitHub (Aug 20, 2021):

@duyhungtnn commented on GitHub (Aug 20, 2021): ![Screen Shot 2021-08-20 at 10 01 22](https://user-images.githubusercontent.com/2597710/130172648-62cab41e-3a4f-41f1-97c8-684c8bceaeb2.png)

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@crossworth commented on GitHub (Aug 20, 2021):

Looks like you need more workers. There are 12 active tasks on the default queue, if they are long running tasks and you have 12 total workers, you are out of workers.

You could try using strict queues but if the process is a long running one I dont think it will make a difference, what you need is some kind of queue reservation, maybe start a server with only the default queue with 2 ou 4 workers and use the rest for the fast running tasks.

@crossworth commented on GitHub (Aug 20, 2021): Looks like you need more workers. There are 12 active tasks on the default queue, if they are long running tasks and you have 12 total workers, you are out of workers. ![](https://user-images.githubusercontent.com/2597710/130171114-04ef65a7-1fc5-4caa-9dd5-1ba744da98ff.png) You could try using [strict queues](https://github.com/hibiken/asynq/wiki/Queue-Priority#strict-priority) but if the process is a long running one I dont think it will make a difference, what you need is some kind of queue reservation, maybe start a server with only the default queue with 2 ou 4 workers and use the rest for the fast running tasks.

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@duyhungtnn commented on GitHub (Aug 20, 2021):

thank you for your suggestion. I tried it before and it works.

I just want to follow the document about weighted priority
But in the document when don't use strict queues, and use the default weighted priority here are whats I expected

tasks in critical queue will be processed 60% of the time
tasks in default queue will be processed 30% of the time
tasks in low queue will be processed 10% of the time

It doesn't happen. After 1 hour of waiting but no critical task is processed. I choose to pause the default queue for a second. And everything comes back to normal.

@duyhungtnn commented on GitHub (Aug 20, 2021): thank you for your suggestion. I tried it before and it works. I just want to follow the document about [weighted priority](https://github.com/hibiken/asynq/wiki/Queue-Priority#weighted-priority) But in the document when don't use strict queues, and use the default weighted priority here are whats I expected ``` tasks in critical queue will be processed 60% of the time tasks in default queue will be processed 30% of the time tasks in low queue will be processed 10% of the time ``` It doesn't happen. After 1 hour of waiting but no `critical` task is processed. I choose to pause the `default` queue for a second. And everything comes back to normal.

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@crossworth commented on GitHub (Aug 20, 2021):

This occurs because you have long running tasks and all the workers are blocked.

If each task takes the same amount of time, the weighted priority would be exact as describe, but when you introduce long running tasks and don't have any free worker you must wait the tasks complete, if they are long running it can take awhile.

When you paused the queue, you had enough workers to process the critical tasks.

You could solve this by limiting the number of long running tasks, making them faster or using multiples servers.

Maybe we should implement more QueueModes like:

Weighted (default)
WeightedStrict
WeightedReserved

Where the WeightedReserved would reserve the workers for each queue without using them (on others queues) when idle, this would keep a lot of workers waiting most of the time, but it could be a good solution for your case and maybe others.

@crossworth commented on GitHub (Aug 20, 2021): This occurs because you have long running tasks and all the workers are blocked. If each task takes the same amount of time, the weighted priority would be exact as describe, but when you introduce long running tasks and don't have any free worker you must wait the tasks complete, if they are long running it can take awhile. When you paused the queue, you had enough workers to process the critical tasks. You could solve this by limiting the number of long running tasks, making them faster or using multiples servers. Maybe we should implement more `QueueModes` like: - Weighted (default) - WeightedStrict - WeightedReserved Where the `WeightedReserved` would reserve the workers for each queue without using them (on others queues) when idle, this would keep a lot of workers waiting most of the time, but it could be a good solution for your case and maybe others.

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@duyhungtnn commented on GitHub (Aug 20, 2021):

@crossworth That is a good idea 👍

In my case right after a long-running task finish, (It usually run in 10 minutes)
we actually have a free worker for a short time and it has so many critical tasks out there. It just could not pick some critical tasks but stick with low priority tasks.

@duyhungtnn commented on GitHub (Aug 20, 2021): @crossworth That is a good idea 👍 In my case right after a long-running task finish, (It usually run in 10 minutes) we actually have a free worker for a short time and it has so many `critical` tasks out there. It just could not pick some `critical` tasks but stick with low priority tasks.

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 21, 2021):

@duyhungtnn thank you for raising this issue! It's something we should consider adding support for or at least update the documentation to make this clearer.

@crossworth's explanation is spot-on. Thank you for jumping in quickly to provide explanations!

For more context, the current implementation picks which queue to pull tasks from by calling this queues() function: github.com/hibiken/asynq@421dc584ff/processor.go (L323)

If the StrictPriority field is set, then it'll always query queues in the priority descending order (i.e. higher priority queue first). So in the scenario you described, if you set the StrictPriority field all your workers should process tasks from critical queue if there's any tasks in the queue.

If the StrictPriority field is not set, then the queues function will use queue priorities to generate a list of queues, each queue name happening the number of priority.
For example, with the below queue configuration:

Queues: map[string]int{
    "critical":  6,
    "default":  3,
    "low":  1,
}

it will generate a list

 // critical*6, defaualt*3, low*1
["critical", "critical", "critical", "critical", "critical", "critical", "default", "default", "default", "low"]

and it shuffles it and dedup the list to get the list we use to query the queues.
So that's how you end up with the chance of 60%, 30%, 10% described in the godoc.

Until we implement something to reserve workers for each queue, potential solutions are to

Use StrictPriority OR
Use higher priority number for the "critical" queue
Dedicate one or more server just to pull tasks from the "critical" queue

Let me know if you have any questions! We'll consider adding support for reserving workers for each queue.

@hibiken commented on GitHub (Aug 21, 2021): @duyhungtnn thank you for raising this issue! It's something we should consider adding support for or at least update the documentation to make this clearer. @crossworth's explanation is spot-on. Thank you for jumping in quickly to provide explanations! For more context, the current implementation picks which queue to pull tasks from by calling this `queues()` function: https://github.com/hibiken/asynq/blob/421dc584ffa4fab4ea8ced5546b9234d2dfedd82/processor.go#L323 If the `StrictPriority` field is set, then it'll **always** query queues in the priority descending order (i.e. higher priority queue first). So in the scenario you described, if you set the `StrictPriority` field all your workers should process tasks from critical queue if there's any tasks in the queue. If the `StrictPriority` field is *not* set, then the `queues` function will use queue priorities to generate a list of queues, each queue name happening the number of priority. For example, with the below queue configuration: ``` Queues: map[string]int{ "critical": 6, "default": 3, "low": 1, } ``` it will generate a list ```go // critical*6, defaualt*3, low*1 ["critical", "critical", "critical", "critical", "critical", "critical", "default", "default", "default", "low"] ``` and it shuffles it and dedup the list to get the list we use to query the queues. So that's how you end up with the chance of 60%, 30%, 10% described in the godoc. Until we implement something to reserve workers for each queue, potential solutions are to - Use `StrictPriority` OR - Use higher priority number for the "critical" queue - Dedicate one or more server just to pull tasks from the "critical" queue Let me know if you have any questions! We'll consider adding support for *reserving* workers for each queue.

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@duyhungtnn commented on GitHub (Aug 22, 2021):

@hibiken I think I have a little misunderstanding in the documentation.
After a day of running, the chances of any queue are true as shown in the Docs. I think it is good.

I fixed this problem by setting up more servers, making sure the critical task is always handled first.

Some servers have the StrictPriority ON.
Some servers only listen on the `critical' queue.

So I think reserving workers could be an ideal feature.

@duyhungtnn commented on GitHub (Aug 22, 2021): @hibiken I think I have a little misunderstanding in the documentation. After a day of running, the chances of any queue are true as shown in the Docs. I think it is good. I fixed this problem by setting up more servers, making sure the ` critical ` task is always handled first. - Some servers have the `StrictPriority` ON. - Some servers only listen on the `critical' queue. So I think `reserving workers` could be an ideal feature.

kerem commented

2026-03-07 22:06:29 +03:00

Author

Owner

@hibiken commented on GitHub (Aug 23, 2021):

@duyhungtnn Glad it worked out. These are great bug-reports/feature-requests. Thank you for reporting & keep 'em coming!

@hibiken commented on GitHub (Aug 23, 2021): @duyhungtnn Glad it worked out. These are great bug-reports/feature-requests. Thank you for reporting & keep 'em coming!

Rows
Columns

[GH-ISSUE #314] [BUG] The high priority queue is process very slow compare to lower queue #1146