[GH-ISSUE #901] [BUG] After manually deleted redis keys, some of the features like metrics, inspector aren't working properly

kerem commented

2026-03-07 22:09:43 +03:00

Owner

Originally created by @sailesh2341 on GitHub (Jul 10, 2024).
Original GitHub issue: https://github.com/hibiken/asynq/issues/901

Originally assigned to: @hibiken on GitHub.

Describe the bug
I have manually deleted one of the key from redis DB to remove one of the task under archive state from queue. After doing this asynqmon metrics stopped working and Inspector methods aren't working to get info of tasks/queue.
The error I'm getting both for metrics and Inspector methods are as follows:

asynq UNKNOWN: UNKNOWN: redis eval error: ERR Error running script (call to f_0c80fc1868e3aae205155ae2017ce995a3a7cec0): @user_script:30: user_script:30: attempt to perform arithmetic on local 'bytes' (a boolean value)

To Reproduce
I've tried many ways to resolve this error:

Restarted the application to restart the asynq server and other instances.
Tried to manually insert the deleted key into redis DB again.
Tried moving some task to archive state and back to pending state again so that the lost keys can be recreated.

Nothing of these approaches worked. Since this issue was on production environment , i cannot flush all the redis DB data nor i cannot take any risky actions .
Has anyone else faced the same issue?
Can someone please tell me how do we resolve this issue?

Originally created by @sailesh2341 on GitHub (Jul 10, 2024). Original GitHub issue: https://github.com/hibiken/asynq/issues/901 Originally assigned to: @hibiken on GitHub. **Describe the bug** I have manually deleted one of the key from redis DB to remove one of the task under archive state from queue. After doing this asynqmon metrics stopped working and Inspector methods aren't working to get info of tasks/queue. The error I'm getting both for metrics and Inspector methods are as follows: asynq UNKNOWN: UNKNOWN: redis eval error: ERR Error running script (call to f_0c80fc1868e3aae205155ae2017ce995a3a7cec0): @user_script:30: user_script:30: attempt to perform arithmetic on local 'bytes' (a boolean value) **To Reproduce** I've tried many ways to resolve this error: 1. Restarted the application to restart the asynq server and other instances. 2. Tried to manually insert the deleted key into redis DB again. 3. Tried moving some task to archive state and back to pending state again so that the lost keys can be recreated. Nothing of these approaches worked. Since this issue was on production environment , i cannot flush all the redis DB data nor i cannot take any risky actions . Has anyone else faced the same issue? Can someone please tell me how do we resolve this issue?

kerem added the

bug

investigate

needs-more-info

labels

2026-03-07 22:09:43 +03:00

kerem commented

2026-03-07 22:09:44 +03:00

Author

Owner

@shchiv commented on GitHub (Mar 18, 2025):

@hibiken @kamikazechaser could you please look into this issue?

@shchiv commented on GitHub (Mar 18, 2025): @hibiken @kamikazechaser could you please look into this issue?

kerem commented

2026-03-07 22:09:44 +03:00

Author

Owner

@AvilovAlex commented on GitHub (Apr 30, 2025):

same problem!

@AvilovAlex commented on GitHub (Apr 30, 2025): same problem!

kerem commented

2026-03-07 22:09:44 +03:00

Author

Owner

@kamikazechaser commented on GitHub (May 5, 2025):

@shchiv @AvilovAlex @sailesh2341 Do you recall the key that you deleted?

@kamikazechaser commented on GitHub (May 5, 2025): @shchiv @AvilovAlex @sailesh2341 Do you recall the key that you deleted?

kerem commented

2026-03-07 22:09:45 +03:00

Author

Owner

@shchiv commented on GitHub (May 5, 2025):

@kamikazechaser, honestly can't say exactly which one

But the issue happened after the update to the latest version, in which the keys format was changed (not sure that's the root cause), and got a lot of redis eval error when trying to use the inspector and GetQueueInfo, somewhere inside it is running eval

@shchiv commented on GitHub (May 5, 2025): @kamikazechaser, honestly can't say exactly which one But the issue happened after the update to the latest version, in which the keys format was changed (not sure that's the root cause), and got a lot of `redis eval error` when trying to use the inspector and GetQueueInfo, somewhere inside it is running eval

kerem commented

2026-03-07 22:09:45 +03:00

Author

Owner

@shchiv commented on GitHub (Jun 10, 2025):

hey @kamikazechaser do you have any updates ?
Could you please help clarify whether the repository is under development and if you are the main developer?

@shchiv commented on GitHub (Jun 10, 2025): hey @kamikazechaser do you have any updates ? Could you please help clarify whether the repository is under development and if you are the main developer?

kerem commented

2026-03-07 22:09:45 +03:00

Author

Owner

@ahmadaidin commented on GitHub (Jan 8, 2026):

I think I got the same issue here. the problem is "MEMORY USAGE <key>" returning non number. the script is in ./internal/rdb/inspect.go memoryUsageCmd variable.

I tried to edit some lines by adding conditional if type(bytes) == "number" and it runs well.

here is the edited script

local sample_size = tonumber(ARGV[2])
if sample_size <= 0 then
    return redis.error_reply("sample size must be a positive number")
end
local memusg = 0
for i=1,2 do
    local ids = redis.call("LRANGE", KEYS[i], 0, sample_size - 1)
    local sample_total = 0
    if (table.getn(ids) > 0) then
        for _, id in ipairs(ids) do
            local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
			if type(bytes) == "number" then
				sample_total = sample_total + bytes
			end
        end
        local n = redis.call("LLEN", KEYS[i])
        local avg = sample_total / table.getn(ids)
        memusg = memusg + (avg * n)
    end
    local m = redis.call("MEMORY", "USAGE", KEYS[i])
    if (m) then
        memusg = memusg + m
    end
end
for i=3,6 do
    local ids = redis.call("ZRANGE", KEYS[i], 0, sample_size - 1)
    local sample_total = 0
    if (table.getn(ids) > 0) then
        for _, id in ipairs(ids) do
            local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
			if type(bytes) == "number" then
				sample_total = sample_total + bytes
			end
        end
        local n = redis.call("ZCARD", KEYS[i])
        local avg = sample_total / table.getn(ids)
        memusg = memusg + (avg * n)
    end
    local m = redis.call("MEMORY", "USAGE", KEYS[i])
    if (m) then
        memusg = memusg + m
    end
end
local groups = redis.call("SMEMBERS", KEYS[7])
if table.getn(groups) > 0 then
	local agg_task_count = 0
	local agg_task_sample_total = 0
	local agg_task_sample_size = 0
	for i, gname in ipairs(groups) do
		local group_key = ARGV[4] .. gname
		agg_task_count = agg_task_count + redis.call("ZCARD", group_key)
		if i <= tonumber(ARGV[3]) then
			local ids = redis.call("ZRANGE", group_key, 0, sample_size - 1)
			for _, id in ipairs(ids) do
				local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
				if type(bytes) == "number" then
				    agg_task_sample_total = agg_task_sample_total + bytes
				end
				agg_task_sample_size = agg_task_sample_size + 1
			end
		end
	end
	local avg = agg_task_sample_total / agg_task_sample_size
	memusg = memusg + (avg * agg_task_count)
end
return memusg

can you confirm if this change can solve your issue? @sailesh2341 @AvilovAlex

@ahmadaidin commented on GitHub (Jan 8, 2026): I think I got the same issue here. the problem is `"MEMORY USAGE <key>"` returning non number. the script is in `./internal/rdb/inspect.go` `memoryUsageCmd` variable. I tried to edit some lines by adding conditional `if type(bytes) == "number" ` and it runs well. here is the edited script ```lua local sample_size = tonumber(ARGV[2]) if sample_size <= 0 then return redis.error_reply("sample size must be a positive number") end local memusg = 0 for i=1,2 do local ids = redis.call("LRANGE", KEYS[i], 0, sample_size - 1) local sample_total = 0 if (table.getn(ids) > 0) then for _, id in ipairs(ids) do local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id) if type(bytes) == "number" then sample_total = sample_total + bytes end end local n = redis.call("LLEN", KEYS[i]) local avg = sample_total / table.getn(ids) memusg = memusg + (avg * n) end local m = redis.call("MEMORY", "USAGE", KEYS[i]) if (m) then memusg = memusg + m end end for i=3,6 do local ids = redis.call("ZRANGE", KEYS[i], 0, sample_size - 1) local sample_total = 0 if (table.getn(ids) > 0) then for _, id in ipairs(ids) do local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id) if type(bytes) == "number" then sample_total = sample_total + bytes end end local n = redis.call("ZCARD", KEYS[i]) local avg = sample_total / table.getn(ids) memusg = memusg + (avg * n) end local m = redis.call("MEMORY", "USAGE", KEYS[i]) if (m) then memusg = memusg + m end end local groups = redis.call("SMEMBERS", KEYS[7]) if table.getn(groups) > 0 then local agg_task_count = 0 local agg_task_sample_total = 0 local agg_task_sample_size = 0 for i, gname in ipairs(groups) do local group_key = ARGV[4] .. gname agg_task_count = agg_task_count + redis.call("ZCARD", group_key) if i <= tonumber(ARGV[3]) then local ids = redis.call("ZRANGE", group_key, 0, sample_size - 1) for _, id in ipairs(ids) do local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id) if type(bytes) == "number" then agg_task_sample_total = agg_task_sample_total + bytes end agg_task_sample_size = agg_task_sample_size + 1 end end end local avg = agg_task_sample_total / agg_task_sample_size memusg = memusg + (avg * agg_task_count) end return memusg ``` can you confirm if this change can solve your issue? @sailesh2341 @AvilovAlex

Rows
Columns

[GH-ISSUE #901] [BUG] After manually deleted redis keys, some of the features like metrics, inspector aren't working properly #1462