#190 - [GH-ISSUE #428] System Design Pattern | Messaging App - starred/asynq

kerem commented

2026-03-02 05:19:30 +03:00

Owner

Originally created by @1337-sys on GitHub (Apr 1, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/428

Hi @hibiken,

First of all, thank you for writing lightest message queue in Go.

I'm writing a messaging app. When the QR is read, I start to process messages. I will use asynq for processing history messages and normal message. I guess I have to create an asynq server for each number (instance)?

Message Handler Case:

Number Instance -> PhoneNumber (Start asynq server when read qr)
QR Read -> Handle All History Messages(avarage 2-3 million message) -> Start worker queue (instance_id_history).
Normal Message (Schedule Message) -> Start worker queue (instance_id_normal)

Should I always listen to 2 queues? Or I can stop listening to this queue when the HistoryMessages is finished, right?

I'll be listening to 2000 queue when there are 1000 number(instances). Do you think a redis cluster(sentinel) is necessary to handle this traffic?

What kind of path do you think we should follow with asynq?

Sincerely :)

Originally created by @1337-sys on GitHub (Apr 1, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/428 Hi @hibiken, First of all, thank you for writing lightest message queue in Go. I'm writing a messaging app. When the QR is read, I start to process messages. I will use asynq for processing history messages and normal message. I guess I have to create an asynq server for each number (instance)? Message Handler Case: - Number Instance -> PhoneNumber (Start asynq server when read qr) - QR Read -> Handle All History Messages(avarage 2-3 million message) -> Start worker queue (instance_id_history). - Normal Message (Schedule Message) -> Start worker queue (instance_id_normal) Should I always listen to 2 queues? Or I can stop listening to this queue when the HistoryMessages is finished, right? I'll be listening to 2000 queue when there are 1000 number(instances). Do you think a redis cluster(sentinel) is necessary to handle this traffic? What kind of path do you think we should follow with asynq? Sincerely :)

kerem closed this issue

2026-03-02 05:19:30 +03:00

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@hibiken commented on GitHub (Apr 2, 2022):

@1337-sys thanks for opening an issue.

2000 queues sound too excessive. You'd probably better off limiting the number of queues to much lower number. You can put variety of task types in the same queue.

Use of redis cluster or sentinel is recommended for high availability and reliability (e.g. automatic failover), but comes with its own operational overhead.

Let me know if you have any follow up questions :)

@hibiken commented on GitHub (Apr 2, 2022): @1337-sys thanks for opening an issue. 2000 queues sound too excessive. You'd probably better off limiting the number of queues to much lower number. You can put variety of task types in the same queue. Use of redis cluster or sentinel is recommended for high availability and reliability (e.g. automatic failover), but comes with its own operational overhead. Let me know if you have any follow up questions :)

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@1337-sys commented on GitHub (Apr 2, 2022):

Thank you so much for your reply :)

Currently there are 400 phones. There are 1 publisher and 800 consumers(history and normal per phone). I have no idea how to reduce consumer number. I guess I can also put history messages in the regular message queue, right? I can't think of any other ideas.

Is it possible for me to send messages from all phones from a single queue? Will it be worse performance when a single queue?

queue -> normal_messages
instead of
queue -> normal_messages/:instance_id.

If phone(instance) is not active or no internet connection, put message back in normal_messages queue? Right?

I don't know if this is what I'm looking for -> https://github.com/hibiken/asynq/issues/339

@1337-sys commented on GitHub (Apr 2, 2022): Thank you so much for your reply :) Currently there are 400 phones. There are 1 publisher and 800 consumers(history and normal per phone). I have no idea how to reduce consumer number. I guess I can also put history messages in the regular message queue, right? I can't think of any other ideas. Is it possible for me to send messages from all phones from a single queue? Will it be worse performance when a single queue? - queue -> normal_messages instead of - queue -> normal_messages/:instance_id. If phone(instance) is not active or no internet connection, put message back in normal_messages queue? Right? I don't know if this is what I'm looking for -> https://github.com/hibiken/asynq/issues/339

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@1337-sys commented on GitHub (Apr 5, 2022):

This link didn't help.
XEP-0160: Best Practices for Handling Offline Messages

I'm still open to good ideas :)

@1337-sys commented on GitHub (Apr 5, 2022): This link didn't help. [XEP-0160: Best Practices for Handling Offline Messages](https://xmpp.org/extensions/xep-0160.html) I'm still open to good ideas :)

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@1337-sys commented on GitHub (Apr 11, 2022):

I think this is what I'm looking for. -> https://docs.nats.io/nats-concepts/subjects

@1337-sys commented on GitHub (Apr 11, 2022): I think this is what I'm looking for. -> https://docs.nats.io/nats-concepts/subjects

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@hibiken commented on GitHub (Apr 12, 2022):

We have a feature request that allows server to listen to dynamic queue names #250, and it's in the roadmap of the project.
I'm considering adding support for prefix matching against queue names. So that you can configure server like this:

// This server will pull tasks from queues with prefix "foo/" (e.g. "foo/bar", "foo/baz", etc)
srv := asynq.NewServer(redisConn, asynq.Config{
     Queues: map[string]int{
         "foo/": 1,
     },
     QueueNameMatchMode: asynq.PrefixMatch, // other possible options are ExactMatch(default), RegexMatch, etc
})

Is this something you are requesting?

@hibiken commented on GitHub (Apr 12, 2022): We have a feature request that allows server to listen to dynamic queue names #250, and it's in the roadmap of the project. I'm considering adding support for prefix matching against queue names. So that you can configure server like this: ```go // This server will pull tasks from queues with prefix "foo/" (e.g. "foo/bar", "foo/baz", etc) srv := asynq.NewServer(redisConn, asynq.Config{ Queues: map[string]int{ "foo/": 1, }, QueueNameMatchMode: asynq.PrefixMatch, // other possible options are ExactMatch(default), RegexMatch, etc }) ``` Is this something you are requesting?

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@hibiken commented on GitHub (Apr 14, 2022):

Closing this. Please feel free to post follow up questions if any :)

@hibiken commented on GitHub (Apr 14, 2022): Closing this. Please feel free to post follow up questions if any :)

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@1337-sys commented on GitHub (Apr 15, 2022):

Yeah yeah, this is exactly what I want. I'm so glad. The 30k line project is counting days to move to this queue library. We'll pass on this feature when it done. @hibiken

@1337-sys commented on GitHub (Apr 15, 2022): Yeah yeah, this is exactly what I want. I'm so glad. The 30k line project is counting days to move to this queue library. We'll pass on this feature when it done. @hibiken

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@archit-harness commented on GitHub (Sep 16, 2022):

@hibiken was this feature added to the Library?

@archit-harness commented on GitHub (Sep 16, 2022): @hibiken was this feature added to the Library?

kerem commented

2026-03-02 05:19:31 +03:00

Author

Owner

@1337-sys commented on GitHub (Dec 14, 2022):