[GH-ISSUE #173] [FEATURE REQUEST] Support other message brokers

kerem commented

2026-03-15 19:03:28 +03:00

Owner

Originally created by @toszo on GitHub (Jun 16, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/173

Originally assigned to: @hibiken on GitHub.

Describe the solution you'd like
RabbitMQ as a message broker for tasks.

Describe alternatives you've considered
MQTT broker which would enable IoT usage.

Additional context
Working on a distributed system that part of which will run on the Edge devices.

Originally created by @toszo on GitHub (Jun 16, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/173 Originally assigned to: @hibiken on GitHub. **Describe the solution you'd like** RabbitMQ as a message broker for tasks. **Describe alternatives you've considered** MQTT broker which would enable IoT usage. **Additional context** Working on a distributed system that part of which will run on the Edge devices.

kerem added the

enhancement

idea

labels

2026-03-15 19:03:28 +03:00

kerem commented

2026-03-15 19:03:40 +03:00

Author

Owner

@hibiken commented on GitHub (Jun 17, 2020):

@toszo Thank you for opening this issue!

I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

@hibiken commented on GitHub (Jun 17, 2020): @toszo Thank you for opening this issue! I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

kerem commented

2026-03-15 19:03:45 +03:00

Author

Owner

@zhaolion commented on GitHub (Jul 18, 2020):

😄 Hello, @hibiken. I had use Kafka & Redis as message broker for async tasks system at the same time(an ugly combination, but it's useful), I hope you will consider adding support for Kafka/RabbitMQ :)

@zhaolion commented on GitHub (Jul 18, 2020): 😄 Hello, @hibiken. I had use Kafka & Redis as message broker for async tasks system at the same time(an ugly combination, but it's useful), I hope you will consider adding support for Kafka/RabbitMQ :)

kerem commented

2026-03-15 19:03:50 +03:00

Author

Owner

@zhaolion commented on GitHub (Jul 18, 2020):

@toszo Thank you for opening this issue!

I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

How can I get the solid features plan in your design, Maybe I can contribute some features 🤓

@zhaolion commented on GitHub (Jul 18, 2020): > @toszo Thank you for opening this issue! > > I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :) How can I get the solid features plan in your design, Maybe I can contribute some features 🤓

kerem commented

2026-03-15 19:03:55 +03:00

Author

Owner

@hibiken commented on GitHub (Jul 18, 2020):

@zhaolion thanks for commenting!
I've been exploring Kafka recently (I don't have any experience with it, so I need to learn it first 😄 ). I think it'd be great if asynq can also support Kafka as a message broker.

In terms of features to implement before working on adding support for other message brokers, I'd like to at least resolve these issues:

Periodic Task feature #183
Support Redis Cluster #182

Once I have tackled those issues, I think we can start looking into adding support to Kafka, RabbitMQ, etc

@hibiken commented on GitHub (Jul 18, 2020): @zhaolion thanks for commenting! I've been exploring Kafka recently (I don't have any experience with it, so I need to learn it first 😄 ). I think it'd be great if `asynq` can also support Kafka as a message broker. In terms of features to implement before working on adding support for other message brokers, I'd like to at least resolve these issues: - Periodic Task feature #183 - Support Redis Cluster #182 Once I have tackled those issues, I think we can start looking into adding support to Kafka, RabbitMQ, etc

kerem commented

2026-03-15 19:04:00 +03:00

Author

Owner

@rugwirobaker commented on GitHub (Aug 19, 2021):

Nats would be nice too. The only reason I hesitate with Redis is data persistence with Redis is a bit more involved. I would love to help with nats of course if there is an isolated Store interface.

@rugwirobaker commented on GitHub (Aug 19, 2021): Nats would be nice too. The only reason I hesitate with Redis is data persistence with Redis is a bit more involved. I would love to help with nats of course if there is an isolated Store interface.

kerem commented

2026-03-15 19:04:05 +03:00

Author

Owner

@no-serve-people commented on GitHub (Sep 15, 2021):

maybe rocketmq is better

@no-serve-people commented on GitHub (Sep 15, 2021): maybe rocketmq is better

kerem commented

2026-03-15 19:04:10 +03:00

Author

Owner

@Me1onRind commented on GitHub (Sep 24, 2021):

An idea, other message queues are supported as plugin. Use interface to decoupling from specific storage.

@Me1onRind commented on GitHub (Sep 24, 2021): An idea, other message queues are supported as plugin. Use interface to decoupling from specific storage.

kerem commented

2026-03-15 19:04:15 +03:00

Author

Owner

@danielkurniadi commented on GitHub (Nov 10, 2021):

Hi @hibiken just want to check on the progress of the blocking features that you mentioned above:

Periodic Task feature [FEATURE REQUEST] Recurring scheduled tasks using a CRON schedule #183
Support Redis Cluster Support RedisCluster client #182

I saw as of today, November 2021. I see its already merged into the codebase for asynq. If we haven't had any progress on the external support for other broker backend (.e.g RabbitMQ, Pulsar, RocketMQ, Kafka, or etc.) I can help with it.

What I will do for you, suppose you think its a good time to start or perhaps has not started yet on these features:

I'll survey into other async task-based queue on what their port (interface) / adapter design looks like to abstract and implement different broker backend. I think Machinery is a good example to look into as they have supported Redis, RabbitMQ, and Google Pubsub(?).
Come up with technical design on how to add port (interface) / adapter design and discuss with the team whether its a good design, review thread-safety issues and possible optimisation.
Then discuss the roadmap and plan how to refactor or migrate current solution to include more broker backend based on agreed design in (2).

Let me know if I can help on this part. You can contact me

Email: daniel.thekurniadi@gmail.com or
LinkedIn: https://www.linkedin.com/in/danielkur/

@danielkurniadi commented on GitHub (Nov 10, 2021): Hi @hibiken just want to check on the progress of the blocking features that you mentioned above: * Periodic Task feature [FEATURE REQUEST] Recurring scheduled tasks using a CRON schedule #183 * Support Redis Cluster Support RedisCluster client #182 I saw as of today, November 2021. I see its already merged into the codebase for asynq. If we haven't had any progress on the external support for other broker backend (.e.g RabbitMQ, Pulsar, RocketMQ, Kafka, or etc.) I can help with it. What I will do for you, suppose you think its a good time to start or perhaps has not started yet on these features: 1. I'll survey into other async task-based queue on what their port (interface) / adapter design looks like to abstract and implement different broker backend. I think Machinery is a good example to look into as they have supported Redis, RabbitMQ, and Google Pubsub(?). 2. Come up with technical design on how to add port (interface) / adapter design and discuss with the team whether its a good design, review thread-safety issues and possible optimisation. 3. Then discuss the roadmap and plan how to refactor or migrate current solution to include more broker backend based on agreed design in (2). Let me know if I can help on this part. You can contact me * Email: daniel.thekurniadi@gmail.com or * LinkedIn: https://www.linkedin.com/in/danielkur/

kerem commented

2026-03-15 19:04:20 +03:00

Author

Owner

@hibiken commented on GitHub (Nov 11, 2021):

@iqDF Thank you for checking in!

I'm still hesitant to start working on this yet since I still feel we haven't built out all the features we want to implement with Redis.
For example, these are some of the feature requests I'm interested in tackling before we can say the library is feature complete with Redis as a message broker:

#339
#244

When we add support for other message brokers, it'll be difficult to add new features.

That being said, we can start looking into the design of the broker interface. Currently we do have an interface type (take a look at internal/base.Broker interface) but it's tightly coupled to the current implementation with Redis as a broker.

@hibiken commented on GitHub (Nov 11, 2021): @iqDF Thank you for checking in! I'm still hesitant to start working on this yet since I still feel we haven't built out all the features we want to implement with Redis. For example, these are some of the feature requests I'm interested in tackling before we can say the library is feature complete with Redis as a message broker: - #339 - #244 When we add support for other message brokers, it'll be difficult to add new features. That being said, we can start looking into the design of the broker interface. Currently we do have an interface type (take a look at `internal/base.Broker` interface) but it's tightly coupled to the current implementation with Redis as a broker.

kerem commented

2026-03-15 19:04:25 +03:00

Author

Owner

@danielkurniadi commented on GitHub (Nov 11, 2021):

This blocking features of Redis are still available?

@danielkurniadi commented on GitHub (Nov 11, 2021): This blocking features of Redis are still available?

kerem commented

2026-03-15 19:04:30 +03:00

Author

Owner

@danielkurniadi commented on GitHub (Nov 11, 2021):

Anyway yup I'll take a look into the design of broker interface.

@danielkurniadi commented on GitHub (Nov 11, 2021): Anyway yup I'll take a look into the design of broker interface.

kerem commented

2026-03-15 19:04:36 +03:00

Author

Owner

@shuqingzai commented on GitHub (Sep 20, 2022):

Looking forward to this release, we now use RabbitMQ for queues and asynq for periodic tasks

@shuqingzai commented on GitHub (Sep 20, 2022): Looking forward to this release, we now use RabbitMQ for queues and asynq for periodic tasks

kerem commented

2026-03-15 19:04:41 +03:00

Author

Owner

@vladfr commented on GitHub (Oct 31, 2022):

Hi @hibiken!

I found asynq while looking up Go and Mongo and queues. I write a small app in vue, backed by a REST server in Go and Mongo, and I really don't want to go into hosting Redis just for a few simple queues.

Is there any renewed interest on this issue? I would love to help out and abstract a backend layer. I really like the simplicity of this project, its interface and ease of use, cli and UI.

I'm thinking of MongoDB because it's widely used, good for small projects, and has the features we'd need (tail cursor, capped cols and atomic updates via FindAndModify().

@vladfr commented on GitHub (Oct 31, 2022): Hi @hibiken! I found `asynq` while looking up Go and Mongo and queues. I write a small app in vue, backed by a REST server in Go and Mongo, and I really don't want to go into hosting Redis just for a few simple queues. Is there any renewed interest on this issue? I would love to help out and abstract a backend layer. I really like the simplicity of this project, its interface and ease of use, cli and UI. I'm thinking of MongoDB because it's widely used, good for small projects, and has the features we'd need (tail cursor, capped cols and atomic updates via `FindAndModify()`.

Rows
Columns

[GH-ISSUE #173] [FEATURE REQUEST] Support other message brokers #2079