[GH-ISSUE #423] [QUESTION] Redis active-active #1203

New issue

Closed

opened 2026-03-07 22:07:21 +03:00 by kerem · 2 comments

kerem commented

2026-03-07 22:07:21 +03:00

Owner

Originally created by @thejosk on GitHub (Mar 21, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/423

Originally assigned to: @hibiken on GitHub.

Hi @hibiken,

We have kubernetes clusters located in multiple (2) data centres. To avoid single point of failure, each data centre will run multiple instances of the PeriodicTaskManagers, and based on the retention/unique logic mentioned in (https://github.com/hibiken/asynq/discussions/376), only 1 unique job can be enqueued at a time. But my question is how will the data in redis clusters be replicated across both the data centers. I understand that this is more of a redis question, but I'd like to know your thoughts/suggestions on how to solve this problem. Thanks again for your work on this amazing library.

Originally created by @thejosk on GitHub (Mar 21, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/423 Originally assigned to: @hibiken on GitHub. Hi @hibiken, We have kubernetes clusters located in multiple (2) data centres. To avoid single point of failure, each data centre will run multiple instances of the `PeriodicTaskManager`s, and based on the retention/unique logic mentioned in (https://github.com/hibiken/asynq/discussions/376), only 1 unique job can be enqueued at a time. But my question is how will the data in redis clusters be replicated across both the data centers. I understand that this is more of a redis question, but I'd like to know your thoughts/suggestions on how to solve this problem. Thanks again for your work on this amazing library.

kerem

2026-03-07 22:07:21 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-07 22:07:22 +03:00

Author

Owner

@hibiken commented on GitHub (Mar 22, 2022):

@thejosk thank you for the question!

As mentioned in https://redis.io/topics/replication, Redis uses asynchronous replication with traditional leader-follow replication scheme . So in order to guarantee uniqueness of a task, clients issuing write requests (e.g. Enqueue task) needs to talk to the leader redis instance (which may be geographically distant from the client). Client in this case is your PeriodicTaskManager processes, and they all need to talk to the leader in order to ensure the uniqueness of tasks.

To avoid single point of failure in redis, you may also want to look into using Redis Sentinel or Redis Cluster.

Let me know if you have more thoughts and questions :)

@hibiken commented on GitHub (Mar 22, 2022): @thejosk thank you for the question! As mentioned in https://redis.io/topics/replication, Redis uses asynchronous replication with traditional leader-follow replication scheme . So in order to guarantee uniqueness of a task, clients issuing write requests (e.g. Enqueue task) needs to talk to the leader redis instance (which may be geographically distant from the client). Client in this case is your `PeriodicTaskManager` processes, and they all need to talk to the leader in order to ensure the uniqueness of tasks. To avoid single point of failure in redis, you may also want to look into using Redis Sentinel or Redis Cluster. Let me know if you have more thoughts and questions :)

kerem commented

2026-03-07 22:07:22 +03:00

Author

Owner

@thejosk commented on GitHub (Mar 25, 2022):

Thanks so much @hibiken for your response. I'll reach out if I have more questions. Closing this now. Thanks for your amazing work.

@thejosk commented on GitHub (Mar 25, 2022): Thanks so much @hibiken for your response. I'll reach out if I have more questions. Closing this now. Thanks for your amazing work.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1203

No description provided.

Rows
Columns