starred/asynq
1
0
Fork 0
mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00
Code Issues 782 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #173] [FEATURE REQUEST] Support other message brokers #1069

New issue
Open
opened 2026-03-07 22:05:07 +03:00 by kerem · 13 comments
kerem commented 2026-03-07 22:05:07 +03:00
Owner
Copy link

Originally created by @toszo on GitHub (Jun 16, 2020).
Original GitHub issue: https://github.com/hibiken/asynq/issues/173

Originally assigned to: @hibiken on GitHub.

Describe the solution you'd like
RabbitMQ as a message broker for tasks.

Describe alternatives you've considered
MQTT broker which would enable IoT usage.

Additional context
Working on a distributed system that part of which will run on the Edge devices.

Originally created by @toszo on GitHub (Jun 16, 2020). Original GitHub issue: https://github.com/hibiken/asynq/issues/173 Originally assigned to: @hibiken on GitHub. **Describe the solution you'd like** RabbitMQ as a message broker for tasks. **Describe alternatives you've considered** MQTT broker which would enable IoT usage. **Additional context** Working on a distributed system that part of which will run on the Edge devices.
kerem commented 2026-03-07 22:05:09 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Jun 17, 2020):

@toszo Thank you for opening this issue!

I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

<!-- gh-comment-id:645114308 --> @hibiken commented on GitHub (Jun 17, 2020): @toszo Thank you for opening this issue! I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)
kerem commented 2026-03-07 22:05:09 +03:00
Author
Owner
Copy link

@zhaolion commented on GitHub (Jul 18, 2020):

😄 Hello, @hibiken. I had use Kafka & Redis as message broker for async tasks system at the same time(an ugly combination, but it's useful), I hope you will consider adding support for Kafka/RabbitMQ :)

<!-- gh-comment-id:660445905 --> @zhaolion commented on GitHub (Jul 18, 2020): 😄 Hello, @hibiken. I had use Kafka & Redis as message broker for async tasks system at the same time(an ugly combination, but it's useful), I hope you will consider adding support for Kafka/RabbitMQ :)
kerem commented 2026-03-07 22:05:09 +03:00
Author
Owner
Copy link

@zhaolion commented on GitHub (Jul 18, 2020):

@toszo Thank you for opening this issue!

I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :)

How can I get the solid features plan in your design, Maybe I can contribute some features 🤓

<!-- gh-comment-id:660446298 --> @zhaolion commented on GitHub (Jul 18, 2020): > @toszo Thank you for opening this issue! > > I've thought about adding support for other message brokers and RabbitMQ is one of them. And this is definitely in the project's roadmap. However, I'd like to work on this after we have solid features implemented with Redis as a broker. If you need support for RabbitMQ soon, I'd say it's better to look for other solutions. It'll probably take a few months before I start working on adding support for other message brokers :) How can I get the solid features plan in your design, Maybe I can contribute some features 🤓
kerem commented 2026-03-07 22:05:10 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Jul 18, 2020):

@zhaolion thanks for commenting!
I've been exploring Kafka recently (I don't have any experience with it, so I need to learn it first 😄 ). I think it'd be great if asynq can also support Kafka as a message broker.

In terms of features to implement before working on adding support for other message brokers, I'd like to at least resolve these issues:

  • Periodic Task feature #183
  • Support Redis Cluster #182

Once I have tackled those issues, I think we can start looking into adding support to Kafka, RabbitMQ, etc

<!-- gh-comment-id:660477606 --> @hibiken commented on GitHub (Jul 18, 2020): @zhaolion thanks for commenting! I've been exploring Kafka recently (I don't have any experience with it, so I need to learn it first 😄 ). I think it'd be great if `asynq` can also support Kafka as a message broker. In terms of features to implement before working on adding support for other message brokers, I'd like to at least resolve these issues: - Periodic Task feature #183 - Support Redis Cluster #182 Once I have tackled those issues, I think we can start looking into adding support to Kafka, RabbitMQ, etc
kerem commented 2026-03-07 22:05:10 +03:00
Author
Owner
Copy link

@rugwirobaker commented on GitHub (Aug 19, 2021):

Nats would be nice too. The only reason I hesitate with Redis is data persistence with Redis is a bit more involved. I would love to help with nats of course if there is an isolated Store interface.

<!-- gh-comment-id:901703935 --> @rugwirobaker commented on GitHub (Aug 19, 2021): Nats would be nice too. The only reason I hesitate with Redis is data persistence with Redis is a bit more involved. I would love to help with nats of course if there is an isolated Store interface.
kerem commented 2026-03-07 22:05:10 +03:00
Author
Owner
Copy link

@no-serve-people commented on GitHub (Sep 15, 2021):

maybe rocketmq is better

<!-- gh-comment-id:919641374 --> @no-serve-people commented on GitHub (Sep 15, 2021): maybe rocketmq is better
kerem commented 2026-03-07 22:05:10 +03:00
Author
Owner
Copy link

@Me1onRind commented on GitHub (Sep 24, 2021):

An idea, other message queues are supported as plugin. Use interface to decoupling from specific storage.

<!-- gh-comment-id:926455979 --> @Me1onRind commented on GitHub (Sep 24, 2021): An idea, other message queues are supported as plugin. Use interface to decoupling from specific storage.
kerem commented 2026-03-07 22:05:11 +03:00
Author
Owner
Copy link

@danielkurniadi commented on GitHub (Nov 10, 2021):

Hi @hibiken just want to check on the progress of the blocking features that you mentioned above:

  • Periodic Task feature [FEATURE REQUEST] Recurring scheduled tasks using a CRON schedule #183
  • Support Redis Cluster Support RedisCluster client #182

I saw as of today, November 2021. I see its already merged into the codebase for asynq. If we haven't had any progress on the external support for other broker backend (.e.g RabbitMQ, Pulsar, RocketMQ, Kafka, or etc.) I can help with it.

What I will do for you, suppose you think its a good time to start or perhaps has not started yet on these features:

  1. I'll survey into other async task-based queue on what their port (interface) / adapter design looks like to abstract and implement different broker backend. I think Machinery is a good example to look into as they have supported Redis, RabbitMQ, and Google Pubsub(?).
  2. Come up with technical design on how to add port (interface) / adapter design and discuss with the team whether its a good design, review thread-safety issues and possible optimisation.
  3. Then discuss the roadmap and plan how to refactor or migrate current solution to include more broker backend based on agreed design in (2).

Let me know if I can help on this part. You can contact me

<!-- gh-comment-id:964950148 --> @danielkurniadi commented on GitHub (Nov 10, 2021): Hi @hibiken just want to check on the progress of the blocking features that you mentioned above: * Periodic Task feature [FEATURE REQUEST] Recurring scheduled tasks using a CRON schedule #183 * Support Redis Cluster Support RedisCluster client #182 I saw as of today, November 2021. I see its already merged into the codebase for asynq. If we haven't had any progress on the external support for other broker backend (.e.g RabbitMQ, Pulsar, RocketMQ, Kafka, or etc.) I can help with it. What I will do for you, suppose you think its a good time to start or perhaps has not started yet on these features: 1. I'll survey into other async task-based queue on what their port (interface) / adapter design looks like to abstract and implement different broker backend. I think Machinery is a good example to look into as they have supported Redis, RabbitMQ, and Google Pubsub(?). 2. Come up with technical design on how to add port (interface) / adapter design and discuss with the team whether its a good design, review thread-safety issues and possible optimisation. 3. Then discuss the roadmap and plan how to refactor or migrate current solution to include more broker backend based on agreed design in (2). Let me know if I can help on this part. You can contact me * Email: daniel.thekurniadi@gmail.com or * LinkedIn: https://www.linkedin.com/in/danielkur/
kerem commented 2026-03-07 22:05:11 +03:00
Author
Owner
Copy link

@hibiken commented on GitHub (Nov 11, 2021):

@iqDF Thank you for checking in!

I'm still hesitant to start working on this yet since I still feel we haven't built out all the features we want to implement with Redis.
For example, these are some of the feature requests I'm interested in tackling before we can say the library is feature complete with Redis as a message broker:

When we add support for other message brokers, it'll be difficult to add new features.

That being said, we can start looking into the design of the broker interface. Currently we do have an interface type (take a look at internal/base.Broker interface) but it's tightly coupled to the current implementation with Redis as a broker.

<!-- gh-comment-id:965879927 --> @hibiken commented on GitHub (Nov 11, 2021): @iqDF Thank you for checking in! I'm still hesitant to start working on this yet since I still feel we haven't built out all the features we want to implement with Redis. For example, these are some of the feature requests I'm interested in tackling before we can say the library is feature complete with Redis as a message broker: - #339 - #244 When we add support for other message brokers, it'll be difficult to add new features. That being said, we can start looking into the design of the broker interface. Currently we do have an interface type (take a look at `internal/base.Broker` interface) but it's tightly coupled to the current implementation with Redis as a broker.
kerem commented 2026-03-07 22:05:11 +03:00
Author
Owner
Copy link

@danielkurniadi commented on GitHub (Nov 11, 2021):

This blocking features of Redis are still available?

<!-- gh-comment-id:965892498 --> @danielkurniadi commented on GitHub (Nov 11, 2021): This blocking features of Redis are still available?
kerem commented 2026-03-07 22:05:11 +03:00
Author
Owner
Copy link

@danielkurniadi commented on GitHub (Nov 11, 2021):

Anyway yup I'll take a look into the design of broker interface.

<!-- gh-comment-id:965894230 --> @danielkurniadi commented on GitHub (Nov 11, 2021): Anyway yup I'll take a look into the design of broker interface.
kerem commented 2026-03-07 22:05:12 +03:00
Author
Owner
Copy link

@shuqingzai commented on GitHub (Sep 20, 2022):

Looking forward to this release, we now use RabbitMQ for queues and asynq for periodic tasks

<!-- gh-comment-id:1251922126 --> @shuqingzai commented on GitHub (Sep 20, 2022): Looking forward to this release, we now use RabbitMQ for queues and asynq for periodic tasks
kerem commented 2026-03-07 22:05:12 +03:00
Author
Owner
Copy link

@vladfr commented on GitHub (Oct 31, 2022):

Hi @hibiken!

I found asynq while looking up Go and Mongo and queues. I write a small app in vue, backed by a REST server in Go and Mongo, and I really don't want to go into hosting Redis just for a few simple queues.

Is there any renewed interest on this issue? I would love to help out and abstract a backend layer. I really like the simplicity of this project, its interface and ease of use, cli and UI.

I'm thinking of MongoDB because it's widely used, good for small projects, and has the features we'd need (tail cursor, capped cols and atomic updates via FindAndModify().

<!-- gh-comment-id:1297424526 --> @vladfr commented on GitHub (Oct 31, 2022): Hi @hibiken! I found `asynq` while looking up Go and Mongo and queues. I write a small app in vue, backed by a REST server in Go and Mongo, and I really don't want to go into hosting Redis just for a few simple queues. Is there any renewed interest on this issue? I would love to help out and abstract a backend layer. I really like the simplicity of this project, its interface and ease of use, cli and UI. I'm thinking of MongoDB because it's widely used, good for small projects, and has the features we'd need (tail cursor, capped cols and atomic updates via `FindAndModify()`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
dependabot/github_actions/golangci/golangci-lint-action-9
dependabot/github_actions/codecov/codecov-action-6
dependabot/github_actions/actions/download-artifact-8
dependabot/github_actions/actions/upload-artifact-7
dependabot/github_actions/actions/checkout-6
dependabot/go_modules/tools/github.com/hibiken/asynq-0.26.0
dependabot/go_modules/tools/github.com/redis/go-redis/v9-9.18.0
dependabot/go_modules/github.com/redis/go-redis/v9-9.18.0
dependabot/go_modules/x/github.com/hibiken/asynq-0.26.0
dependabot/go_modules/tools/github.com/mattn/go-runewidth-0.0.23
dependabot/go_modules/x/github.com/redis/go-redis/v9-9.18.0
dependabot/go_modules/golang.org/x/sys-0.43.0
dependabot/go_modules/tools/github.com/prometheus/client_golang-1.23.2
dependabot/go_modules/google.golang.org/protobuf-1.36.11
dependabot/go_modules/tools/github.com/spf13/pflag-1.0.10
dependabot/go_modules/golang.org/x/time-0.15.0
dependabot/go_modules/x/github.com/prometheus/client_golang-1.23.2
sohail/v0.27.0
pr-1100
next
preview-v0.26.0
sohail/recoverer-fix
sohail/unix-stop-signal
sohail/pm-redis-conn-hotfix
sohail/release-v0.25.0
revert-937-dependabot/go_modules/golang.org/x/time-0.7.0
develop
sohail/changelog/v0.25.0
go-versions
sohail/activate-dependabot
revert-491-master
v0.26.0
v0.25.1
v0.25.0
v0.24.1
v0.24.0
v0.23.0
v0.22.1
v0.22.0
v0.21.0
v0.20.0
v0.19.1
v0.19.0
v0.18.6
v0.18.5
v0.18.4
v0.18.3
v0.18.2
v0.18.1
v0.18.0
v0.17.2
v0.17.1
v0.17.0
v0.16.1
v0.16.0
v0.15.0
v0.14.1
v0.14.0
v0.13.1
v0.13.0
v0.12.0
v0.11.0
v0.10.0
v0.10.0.rc1
v0.9.4
v0.9.3
v0.9.2
v0.9.1
v0.9.0
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.1
v0.7.0
v0.6.2
v0.6.1
v0.6.0
v0.5.0
v0.4.0
v0.3.0
v0.2.2
v0.2.1
v0.2.0
v0.1.0
Labels
Clear labels   
CLI

   
bug

   
designing

   
documentation

   
duplicate

   
enhancement

   
good first issue

   
good first issue

   
help wanted

   
idea

   
invalid

   
investigate

   
needs-more-info

   
performance

   
pr-welcome

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
wontfix

   
work in progress

   
work in progress

   
work-around-available
No labels
CLI
bug
designing
documentation
duplicate
enhancement
good first issue
good first issue
help wanted
idea
invalid
investigate
needs-more-info
performance
pr-welcome
pull-request
question
wontfix
work in progress
work in progress
work-around-available
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/asynq#1069
No description provided.