mirror of
https://github.com/hibiken/asynq.git
synced 2026-04-25 23:15:51 +03:00
[GH-ISSUE #764] [BUG] WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND #375
Labels
No labels
CLI
bug
designing
documentation
duplicate
enhancement
good first issue
good first issue
help wanted
idea
invalid
investigate
needs-more-info
performance
pr-welcome
pull-request
question
wontfix
work in progress
work in progress
work-around-available
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/asynq#375
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jeffphp on GitHub (Oct 13, 2023).
Original GitHub issue: https://github.com/hibiken/asynq/issues/764
Originally assigned to: @hibiken on GitHub.
Describe the bug
I have 8 asynq workers on different machines (2 queues x 4 workers) and I regularly have this in the logs. My broker/backend is a unique Redis (7). I am using latest version of asynq (v0.24.1) :
asynq: pid=1 2023/10/13 07:29:56.418053 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:29:56.419020 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:56.423351 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:56.424304 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:56.425348 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:56.426263 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:56.430068 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:56.430850 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:56.431540 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:56.432297 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:00.950383 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:00.951512 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:00.955868 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:00.956701 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:00.957690 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:00.958766 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:32:00.963465 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:32:00.964521 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:32:00.965856 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:32:00.967131 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:29:05.304537 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:29:05.305448 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:05.309008 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:05.310101 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:05.311146 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:05.312563 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:05.316957 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:05.318012 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:05.319084 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:05.320076 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:29:10.319618 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:29:10.321065 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:10.325028 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:10.326608 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:10.328063 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:30:10.329341 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:10.334770 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:10.336214 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:10.337392 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
asynq: pid=1 2023/10/13 07:31:10.338755 WARN: recoverer: could not move task to archive: INTERNAL_ERROR: redis eval error: NOT FOUND
To Reproduce
Steps to reproduce the behavior :
Expected behavior
I would like the logs to be more explicit because I don't known which task is problematic.
Throwing a SkipRetry error on a unique Task seems to corrupt the queue. Ss the task is active but orphaned, it's impossible to delete the queue, except using ZRANGE/LRANGE manual redis queries.
Environment (please complete the following information):
v0.24.1
Update on october 21st
asynqmon shows the following error :
"Error
Could not retrieve queues live data — See the logs for details", but it doesn't mention what logs to watch for details.
Using asynq-cli, I get the following error on"asynq queue inspect low_priority" :
error: UNKNOWN: UNKNOWN: redis eval error: ERR user_script:30: attempt to perform arithmetic on local 'bytes' (a boolean value) script: 0c80fc1868e3aae205155ae2017ce995a3a7cec0, on @user_script:30.
@jeffphp commented on GitHub (Oct 22, 2023):
As a workaround I ended up stopping using asynq.SkipRetry error because it corrupts the queue when used with a Unique ID task.
@kamikazechaser commented on GitHub (Oct 23, 2023):
Could you try if #709 fixes this issue?
@jeffphp commented on GitHub (Oct 23, 2023):
I will definitely do. For the time being, I am trying to reproduce it but it's more elaborate that I thought. I think that another condition is that the worker is being stopped in the meanwhile. I'll let you know.
@bingtianyiyan commented on GitHub (Jan 31, 2024):
i have not use Unique ID，it also have this error when consumer msg handler
@wuwenrufeng commented on GitHub (Feb 29, 2024):
The reason I encountered this situation is because I manually deleted some of the keys of asynq in redis, which caused data inconsistency. For example, ZRANGEBYSCORE asynq:{default}:lease -inf 1709175830 will find empty tasks. In short, it is okay not to manually delete data.
@jackbaron commented on GitHub (Mar 31, 2025):
@wuwenrufeng I have same problem how to solved. You found any solution?
@imamrafii commented on GitHub (Apr 4, 2025):
Do you guys have any solution to clearing orphaned tasks caused by manually deleting data?
@jackbaron commented on GitHub (Apr 4, 2025):
@imamrafii This is solution working for me
``
DEL "asynq:{default}:t:task-id-generate"
ZREM asynq:{default}:task-id-generate
``
@jeffphp commented on GitHub (Apr 4, 2025):
Still not
@jeffphp commented on GitHub (May 26, 2025):
In the end, it happens when one of the worker has a fatal error when processing a task (OOMKilled or Fatal Error). It ends up not sending an acknowledgement and the task is archived. If a lot of tasks get archived this way, it stops responding. If the tasks are manually retried, they become Orphaned.
The only way to get out of this is to make sure that workers don't die (!). If it happens, you can delete the whole queue using :
go install github.com/hibiken/asynq/tools/asynq@latest
And then delete the "ill" queue :
asynq queue rm --force <queue_name>
@jeffphp commented on GitHub (Dec 4, 2025):
I think I found the origin of the problem (in my case): I use a managed Redis and found out that it was killing idle connections after 5 minutes. I had this kind of errors : write tcp 10.244..:36820->206.189..:25061: write: broken pipe
I ended up configuring my Redis client with this:
opt.PoolSize = 20
opt.ConnMaxIdleTime = 3 * time.Minute
Now everything seems to work.
@ncimbaljevic commented on GitHub (Dec 9, 2025):
@jeffphp
DUDE! You just made my agony to go away!
Such a easy and logical thing, however I haven't tried it until I read it here.
Cudos to you Sir!
@jeffphp commented on GitHub (Dec 9, 2025):
You're very welcome, I struggled for month with this too! I think that lots of errors (redis eval, redis broken pipe, etc) are related to this.
By the way, asynq.RedisClientOpt{} should definitely provide this ConnMaxIdleTime option with a default value lower than 5 minutes (which seems to be the standard value). I had to use the UniversalClient to make it work properly.
cc @hibiken