starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00

[GH-ISSUE #300] Question: is it possible for multiple "servers" to "forward" the same messages #123

New issue

Closed

opened 2026-03-02 05:18:52 +03:00 by kerem · 4 comments

kerem commented

2026-03-02 05:18:52 +03:00

Owner

Originally created by @seanyu4296 on GitHub (Jul 25, 2021).
Original GitHub issue: https://github.com/hibiken/asynq/issues/300

Originally assigned to: @hibiken on GitHub.

Hi, @hibiken

First off, awesome lib! I'm trying to study and use your library. I'm pretty new to go and redis too.

So, Can I ask is it possible for multiple "servers" to "forward" the same messages given the code below?

var forwardCmd = redis.NewScript(`
local ids = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1], "LIMIT", 0, 100)
for _, id in ipairs(ids) do
	redis.call("LPUSH", KEYS[2], id)
	redis.call("ZREM", KEYS[1], id)
	redis.call("HSET", ARGV[2] .. id, "state", "pending")
end
return table.getn(ids)`)

wouldn't 2 servers querying from the same queues end up querying the same items if they are started almost at the same time?
Wondering also if you have some docs explaining the rough architecture of your lib, it will be tremendously helpful

Thanks,

Originally created by @seanyu4296 on GitHub (Jul 25, 2021). Original GitHub issue: https://github.com/hibiken/asynq/issues/300 Originally assigned to: @hibiken on GitHub. Hi, @hibiken First off, awesome lib! I'm trying to study and use your library. I'm pretty new to go and redis too. So, Can I ask is it possible for multiple "servers" to "forward" the same messages given the code below? ```golang var forwardCmd = redis.NewScript(` local ids = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1], "LIMIT", 0, 100) for _, id in ipairs(ids) do redis.call("LPUSH", KEYS[2], id) redis.call("ZREM", KEYS[1], id) redis.call("HSET", ARGV[2] .. id, "state", "pending") end return table.getn(ids)`) ``` wouldn't 2 servers querying from the same queues end up querying the same items if they are started almost at the same time? Wondering also if you have some docs explaining the rough architecture of your lib, it will be tremendously helpful Thanks,

kerem

2026-03-02 05:18:52 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-02 05:18:53 +03:00

Author

Owner

@hibiken commented on GitHub (Jul 25, 2021):

Hi @seanyu4296 thank you for the question!

Redis is single-threaded from the POV of commands execution.
Please take a look at this Atomicity of scripts in redis doc.

quotes from redis documentation:

Redis uses the same Lua interpreter to run all the commands. Also Redis guarantees that a script is executed in an atomic way: no other script or Redis command will be executed while a script is being executed. This semantic is similar to the one of MULTI / EXEC. From the point of view of all the other clients the effects of a script are either still not visible or already completed.
However this also means that executing slow scripts is not a good idea. It is not hard to create fast scripts, as the script overhead is very low, but if you are going to use slow scripts you should be aware that while the script is running no other client can execute commands.

Second part about slow scripts explains why we are currently dealing with 100 tasks at a time (to avoid running a slow script which blocks other commands from executing).

I'll document the design once the library is stable (i.e. version 1.0 release).

@hibiken commented on GitHub (Jul 25, 2021): Hi @seanyu4296 thank you for the question! Redis is single-threaded from the POV of commands execution. Please take a look at this [Atomicity of scripts](https://redis.io/commands/eval#atomicity-of-scripts) in redis doc. quotes from redis documentation: > Redis uses the same Lua interpreter to run all the commands. Also Redis guarantees that a script is executed in an atomic way: no other script or Redis command will be executed while a script is being executed. This semantic is similar to the one of MULTI / EXEC. From the point of view of all the other clients the effects of a script are either still not visible or already completed. > However this also means that executing slow scripts is not a good idea. It is not hard to create fast scripts, as the script overhead is very low, but if you are going to use slow scripts you should be aware that while the script is running no other client can execute commands. Second part about slow scripts explains why we are currently dealing with 100 tasks at a time (to avoid running a slow script which blocks other commands from executing). I'll document the design once the library is stable (i.e. version 1.0 release).

kerem commented

2026-03-02 05:18:53 +03:00

Author

Owner

@seanyu4296 commented on GitHub (Jul 26, 2021):

gotcha thanks for the quick, succinct, and clear explanation! @hibiken

Do you have a timeline or roadmap to version 1.0 release? 😄

@seanyu4296 commented on GitHub (Jul 26, 2021): gotcha thanks for the quick, succinct, and clear explanation! @hibiken Do you have a timeline or roadmap to version 1.0 release? 😄

kerem commented

2026-03-02 05:18:53 +03:00

Author

Owner

@hibiken commented on GitHub (Jul 26, 2021):

I have this milestone https://github.com/hibiken/asynq/milestone/2

The roadmap is not set in stone, it may change in the future.

@hibiken commented on GitHub (Jul 26, 2021): I have this milestone https://github.com/hibiken/asynq/milestone/2 The roadmap is not set in stone, it may change in the future.

kerem commented

2026-03-02 05:18:53 +03:00

Author

Owner

@seanyu4296 commented on GitHub (Jul 26, 2021):

Awesome stuff thanks! @hibiken

@seanyu4296 commented on GitHub (Jul 26, 2021): Awesome stuff thanks! @hibiken

kerem referenced this issue

2026-03-07 22:04:45 +03:00

[GH-ISSUE #123] Add Wiki page for Middleware, ServeMux, Timeout, Deadline. #1049

kerem referenced this issue

2026-03-15 18:58:18 +03:00

[GH-ISSUE #123] Add Wiki page for Middleware, ServeMux, Timeout, Deadline. #2059

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#123

No description provided.

Rows
Columns