starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00

[GH-ISSUE #352] Is batch tasks possible? #2178

New issue

Closed

opened 2026-03-15 19:34:16 +03:00 by kerem · 2 comments

kerem commented

2026-03-15 19:34:16 +03:00

Owner

Originally created by @1337-sys on GitHub (Nov 19, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/352

Thank you very much for writing such a wonderful queue library.

I want to use it in production but I was wondering if batch processing is possible. I will bulk insert the data waiting in the queue to the db.

Finally, I would like to ask you something based on your experience.

A user has many notifications. I am writing these notifications to the database. But my db connection pool fills up immediately. Is it appropriate to open a queue for each user or to put all notifications in one queue?

queue -> uniqueUserID_notifications(per user)
queue -> notifications

Which would be more suitable? Can I delete notifications from someone with user id x when there is only one queue?

Originally created by @1337-sys on GitHub (Nov 19, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/352 Thank you very much for writing such a wonderful queue library. 1. I want to use it in production but I was wondering if batch processing is possible. I will bulk insert the data waiting in the queue to the db. Finally, I would like to ask you something based on your experience. 2. A user has many notifications. I am writing these notifications to the database. But my db connection pool fills up immediately. Is it appropriate to open a queue for each user or to put all notifications in one queue? queue -> uniqueUserID_notifications(per user) queue -> notifications Which would be more suitable? Can I delete notifications from someone with user id x when there is only one queue?

kerem closed this issue

2026-03-15 19:34:22 +03:00

kerem commented

2026-03-15 19:34:27 +03:00

Author

Owner

@hibiken commented on GitHub (Nov 20, 2021):

@1337-sys Thank you for opening an issue.

Let me make sure that I understood your questions correctly:

I want to use it in production but I was wondering if batch processing is possible. I will bulk insert the data waiting in the queue to the db.

To use an example, if you enqueue task A, B and C to a queue. You want to handle these three tasks as a unit (e.g. in batch) instead of handling them individually? In other words, you want to group these tasks into one task that gets delivered to your Handler? If this is what you meant, there is an open feature-request: #339 and I will be working on it once I have a good design.

To answer the second question:
It's currently not possible to dynamically define which queues to consume tasks from in server's Config. A potential feature request would be to add support for prefix based queue matching when specifying the queues in server config.

Can I delete notifications from someone with user id x when there is only one queue?

You could use Inspector.DeleteTask to delete a task with queue name and task ID, but I'm not quite sure if this is what you are looking for.

If you could provide more context or examples, I can answer better to these questions :)

@hibiken commented on GitHub (Nov 20, 2021): @1337-sys Thank you for opening an issue. Let me make sure that I understood your questions correctly: > I want to use it in production but I was wondering if batch processing is possible. I will bulk insert the data waiting in the queue to the db. To use an example, if you enqueue task A, B and C to a queue. You want to handle these three tasks as a unit (e.g. in batch) instead of handling them individually? In other words, you want to group these tasks into one task that gets delivered to your `Handler`? If this is what you meant, there is an open feature-request: #339 and I will be working on it once I have a good design. To answer the second question: It's currently not possible to dynamically define which queues to consume tasks from in server's Config. A potential feature request would be to add support for prefix based queue matching when specifying the queues in server config. > Can I delete notifications from someone with user id x when there is only one queue? You could use `Inspector.DeleteTask` to delete a task with queue name and task ID, but I'm not quite sure if this is what you are looking for. If you could provide more context or examples, I can answer better to these questions :)

kerem commented

2026-03-15 19:34:32 +03:00

Author

Owner

@hibiken commented on GitHub (Jan 16, 2022):

Duplicate of #339

@hibiken commented on GitHub (Jan 16, 2022): Duplicate of #339

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2178

No description provided.

Rows
Columns