starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00

[GH-ISSUE #766] [FEATURE REQUEST] Separate Consumer from Server #2399

New issue

Open

opened 2026-03-15 20:20:50 +03:00 by kerem · 4 comments

kerem commented

2026-03-15 20:20:50 +03:00

Owner

Originally created by @sujit-baniya on GitHub (Oct 15, 2023).
Original GitHub issue: https://github.com/hibiken/asynq/issues/766

Originally assigned to: @hibiken on GitHub.

Is your feature request related to a problem? Please describe.
Currently running multiple server for consuming queues also runs multiple server related processes like janitor, scheduler and so on.
Separate Consumer will be entirely use for processing data and nothing else.

Originally created by @sujit-baniya on GitHub (Oct 15, 2023). Original GitHub issue: https://github.com/hibiken/asynq/issues/766 Originally assigned to: @hibiken on GitHub. **Is your feature request related to a problem? Please describe.** Currently running multiple server for consuming queues also runs multiple server related processes like janitor, scheduler and so on. Separate Consumer will be entirely use for processing data and nothing else.

kerem added the

enhancement

idea

labels

2026-03-15 20:20:50 +03:00

kerem commented

2026-03-15 20:21:01 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Oct 19, 2023):

Isn't the scheduler already decoupled and needs to be explicitly started?

The janitor and other health check goroutines are core to cleaning up things and keeping everything in check.

@kamikazechaser commented on GitHub (Oct 19, 2023): Isn't the scheduler already decoupled and needs to be explicitly started? The janitor and other health check goroutines are core to cleaning up things and keeping everything in check.

kerem commented

2026-03-15 20:21:06 +03:00

Author

Owner

@sujit-baniya commented on GitHub (Oct 19, 2023):

@kamikazechaser Scheduler is different as per implementation. I'm referring to separating to two services:

Server: would manage jantor, healtch checks of consumers (also could act as consumer)
Consumers: wait and consume tasks that are assigned with queues.

This way consumer would be lightweight to implement for specific task.

What is currently happening?

I'm creating two servers (consumers) to handle tasks on multiple machines. Now both servers do janitor, health checks and other stuffs that IMO should be centralized.

This mechanism is similar to rabbitmq or nats.

@sujit-baniya commented on GitHub (Oct 19, 2023): @kamikazechaser Scheduler is different as per implementation. I'm referring to separating to two services: 1) Server: would manage jantor, healtch checks of consumers (also could act as consumer) 2) Consumers: wait and consume tasks that are assigned with queues. This way consumer would be lightweight to implement for specific task. **What is currently happening?** I'm creating two servers (consumers) to handle tasks on multiple machines. Now both servers do janitor, health checks and other stuffs that IMO should be centralized. This mechanism is similar to rabbitmq or nats.

kerem commented

2026-03-15 20:21:11 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Oct 20, 2023):

I see, one server could start the additional goroutines to perform cleanups, health checks e.t.c while the other instances could just be pulling from the queue.

While it does make sense to have finer control, I am not sure the overhead is enough to warrant a potentially huge refactor. But if you are willing to implement this, you could start an RFC spec here and get some feedback.

@kamikazechaser commented on GitHub (Oct 20, 2023): I see, one server could start the additional goroutines to perform cleanups, health checks e.t.c while the other instances could just be pulling from the queue. While it does make sense to have finer control, I am not sure the overhead is enough to warrant a potentially huge refactor. But if you are willing to implement this, you could start an RFC spec here and get some feedback.

kerem commented

2026-03-15 20:21:17 +03:00

Author

Owner

@lukebarton commented on GitHub (Aug 19, 2024):

One of the most common use case for queues is to distribute the work across compute instances, so it's a little unfortunate that asynq only facilitates distribution across multiple threads in a single compute instance.

@lukebarton commented on GitHub (Aug 19, 2024): One of the most common use case for queues is to _distribute_ the work across compute instances, so it's a little unfortunate that asynq only facilitates distribution across multiple threads in a single compute instance.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2399

No description provided.

Rows
Columns