[GH-ISSUE #175] Create Dynamic Queues? #56

New issue

Closed

opened 2026-03-02 05:18:18 +03:00 by kerem · 4 comments

kerem commented

2026-03-02 05:18:18 +03:00

Owner

Originally created by @muku2211 on GitHub (Jun 24, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/175

Hi, I need to create queues dynamically on the fly and have the workers consume from these queues. Currently I'm able to create new queues by just passing the name of the new queue using asynq.Queue but the workers do not pick them up. Is there a way to subscibe running workers to the newly created queue?

Originally created by @muku2211 on GitHub (Jun 24, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/175 Hi, I need to create queues dynamically on the fly and have the workers consume from these queues. Currently I'm able to create new queues by just passing the name of the new queue using `asynq.Queue` but the workers do not pick them up. Is there a way to subscibe running workers to the newly created queue?

kerem closed this issue

2026-03-02 05:18:18 +03:00

kerem commented

2026-03-02 05:18:19 +03:00

Author

Owner

@hibiken commented on GitHub (Jun 24, 2020):

@muku2211 Thank you for opening this issue!
With the current API, we need to specify the list of queues when we initialize asynq.Server.
It is possible to add an ability to consume all queues, but I'd like to understand your use case a bit more before adding that option.

Is it possible to enqueue all your tasks to the default queue without dynamically creating named queues?
If not, would you mind explaining why you need to create queues dynamically?

@hibiken commented on GitHub (Jun 24, 2020): @muku2211 Thank you for opening this issue! With the current API, we need to specify the list of queues when we initialize `asynq.Server`. It is possible to add an ability to consume all queues, but I'd like to understand your use case a bit more before adding that option. Is it possible to enqueue all your tasks to the default queue without dynamically creating named queues? If not, would you mind explaining why you need to create queues dynamically?

kerem commented

2026-03-02 05:18:19 +03:00

Author

Owner

@muku2211 commented on GitHub (Jun 25, 2020):

@hibiken Thanks for the quick response!
Our use case is to create and run outbound dialling campaigns and each campaign has to be run during specific times. For eg. if there are two campaigns A & B, A should be run from 12:00pm - 3:00pm whereas B should be run from 6:00pm - 8:00pm. This can be achieved with asynq's pause/unpause functionality.

Also, we want to make sure that there is fairness among campaigns. For eg. If Campaign A has 5 calls [1,2,3,4,5] and Campaign B has 3 calls [1.2.3], we want to make sure that the calls are processes in this order: A1, B1, A2, B2, A3, B3, A4, A5.
So the idea was to create a queue dynamically for each new Campaign (same priority) and have the workers automatically consume from these newly created queues.

@muku2211 commented on GitHub (Jun 25, 2020): @hibiken Thanks for the quick response! Our use case is to create and run outbound dialling campaigns and each campaign has to be run during specific times. For eg. if there are two campaigns A & B, A should be run from 12:00pm - 3:00pm whereas B should be run from 6:00pm - 8:00pm. This can be achieved with asynq's pause/unpause functionality. Also, we want to make sure that there is fairness among campaigns. For eg. If Campaign A has 5 calls [1,2,3,4,5] and Campaign B has 3 calls [1.2.3], we want to make sure that the calls are processes in this order: A1, B1, A2, B2, A3, B3, A4, A5. So the idea was to create a queue dynamically for each new Campaign (same priority) and have the workers automatically consume from these newly created queues.

kerem commented

2026-03-02 05:18:19 +03:00

Author

Owner

@hibiken commented on GitHub (Jun 25, 2020):

I see. Thank you for that detail.

From what you shared, I think you'd be able to achieve the same effect using EnqueueAt and providing the time each task should be processed.

For example, if you have tasks A1, B1, A2, B2, A3, B3, A4, A5 and these need to be processed in this order, then you can
schedule these tasks with EnqueueAt.

// Schedule each task with one minute increment.
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 0, 0, 0, time.UTC), taskA1)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 1, 0, 0, time.UTC), taskB1)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 2, 0, 0, time.UTC), taskA2)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 3, 0, 0, time.UTC), taskB2)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 4, 0, 0, time.UTC), taskA3)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 5, 0, 0, time.UTC), taskB3)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 6, 0, 0, time.UTC), taskA4)
client.EnqueueAt(time.Date((2020, time.June, 25, 18, 7, 0, 0, time.UTC), taskA5)

This way you don't have to create separate queues and the task will be processed in the intended order.

Let me know if this works!

@hibiken commented on GitHub (Jun 25, 2020): I see. Thank you for that detail. From what you shared, I think you'd be able to achieve the same effect using `EnqueueAt` and providing the time each task should be processed. For example, if you have tasks A1, B1, A2, B2, A3, B3, A4, A5 and these need to be processed in this order, then you can schedule these tasks with `EnqueueAt`. ```go // Schedule each task with one minute increment. client.EnqueueAt(time.Date((2020, time.June, 25, 18, 0, 0, 0, time.UTC), taskA1) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 1, 0, 0, time.UTC), taskB1) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 2, 0, 0, time.UTC), taskA2) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 3, 0, 0, time.UTC), taskB2) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 4, 0, 0, time.UTC), taskA3) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 5, 0, 0, time.UTC), taskB3) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 6, 0, 0, time.UTC), taskA4) client.EnqueueAt(time.Date((2020, time.June, 25, 18, 7, 0, 0, time.UTC), taskA5) ``` This way you don't have to create separate queues and the task will be processed in the intended order. Let me know if this works!

kerem commented

2026-03-02 05:18:19 +03:00

Author

Owner

@hibiken commented on GitHub (Jul 8, 2020):

Closing this for now.

@hibiken commented on GitHub (Jul 8, 2020): Closing this for now.

kerem referenced this issue

2026-03-02 05:54:05 +03:00

[PR #56] [MERGED] Add syncer to retry failed requests to redis #573

kerem referenced this issue

2026-03-07 22:11:25 +03:00