starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-26 07:25:56 +03:00

[GH-ISSUE #542] [FEATURE REQUEST] How can I specify queue concurrency (separate from worker concurrency) #2282

New issue

Open

opened 2026-03-15 19:56:49 +03:00 by kerem · 2 comments

kerem commented

2026-03-15 19:56:49 +03:00

Owner

Originally created by @halorium on GitHub (Sep 20, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/542

Originally assigned to: @hibiken on GitHub.

It's critical that I'm able to specify the following:

The total number of simultaneous workers that can run at any given time. (e.g. 1 per each cpu core for a total of 4)
The total number of simultaneous tasks that can run at any given time (per queue).
Given the following queues:
queue_one
queue_two
Only one task from queue_one can run at any given time.
Any number of tasks from queue_two can run at any given time.
e.g.
queue_one [<task 1>, <task 3>, <task 5>]
queue_two [<task 2>, <task 4>, <task 6>]

worker_1 processing <task 1> from queue_one
worker_2 processing <task 2> from queue_two
worker_3 processing <task 4> from queue_two
worker_4 processing <task 6> from queue_two

The point here is that a total of 4 tasks can be processed simultaneously as per the "worker concurrency count" but only 1 task from queue_one can be processed simultaneously as per the "queue concurrency count".

This is a critical feature for me as there are types of tasks that would interfere with each other if they were run at the same time. Therefore they would be in the same queue and that queue would have a concurrency count of 1. aka only one of those tasks can run at a time.

Please let me know if this ability exists in this package. Thank you.

Originally created by @halorium on GitHub (Sep 20, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/542 Originally assigned to: @hibiken on GitHub. It's critical that I'm able to specify the following: * The total number of simultaneous workers that can run at any given time. (e.g. 1 per each cpu core for a total of 4) * The total number of simultaneous tasks that can run at any given time (per queue). Given the following queues: * queue_one * queue_two Only one task from queue_one can run at any given time. Any number of tasks from queue_two can run at any given time. e.g. queue_one [<task 1>, <task 3>, <task 5>] queue_two [<task 2>, <task 4>, <task 6>] worker_1 processing <task 1> from queue_one worker_2 processing <task 2> from queue_two worker_3 processing <task 4> from queue_two worker_4 processing <task 6> from queue_two The point here is that a total of 4 tasks can be processed simultaneously as per the "worker concurrency count" but only 1 task from queue_one can be processed simultaneously as per the "queue concurrency count". This is a critical feature for me as there are types of tasks that would interfere with each other if they were run at the same time. Therefore they would be in the same queue and that queue would have a concurrency count of 1. aka only one of those tasks can run at a time. Please let me know if this ability exists in this package. Thank you.

kerem added the

enhancement

label

2026-03-15 19:56:49 +03:00

kerem commented

2026-03-15 19:57:00 +03:00

Author

Owner

@TheStu commented on GitHub (Nov 12, 2022):

This is more or less what I'm looking to accomplish as well. I have a large amount of jobs to be processed, about 20 different types. But one type can only be performed once per 2 seconds. I don't want 10 workers all stuck processing these rate limited jobs when there's ~10k other non-rate limited jobs to process, but I do want to perform ~1 rate limited job per 2 seconds so that they get done as quickly as possible.

The simplest solution seems to be to have 1 worker that works on rate limited jobs, and the rest of the workers process the non-rate limited jobs. Alternatively, a rate limit or timeout on a given queue.

@TheStu commented on GitHub (Nov 12, 2022): This is more or less what I'm looking to accomplish as well. I have a large amount of jobs to be processed, about 20 different types. But one type can only be performed once per 2 seconds. I don't want 10 workers all stuck processing these rate limited jobs when there's ~10k other non-rate limited jobs to process, but I do want to perform ~1 rate limited job per 2 seconds so that they get done as quickly as possible. The simplest solution seems to be to have 1 worker that works on rate limited jobs, and the rest of the workers process the non-rate limited jobs. Alternatively, a rate limit or timeout on a given queue.

kerem commented

2026-03-15 19:57:05 +03:00

Author

Owner

@BrandSnob commented on GitHub (Jan 17, 2024):

The following merged PR can help you:
https://github.com/hibiken/asynq/pull/336

However, I also would like a generic feature for this purpose.

@BrandSnob commented on GitHub (Jan 17, 2024): The following merged PR can help you: https://github.com/hibiken/asynq/pull/336 However, I also would like a generic feature for this purpose.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2282

No description provided.

Rows
Columns