starred/asynq
1
0
Fork 0
mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00
Code Issues 782 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1051] [BUG] Tasks stuck in active state #502

New issue
Open
opened 2026-03-02 05:37:54 +03:00 by kerem · 4 comments
kerem commented 2026-03-02 05:37:54 +03:00
Owner
Copy link

Originally created by @asadzeynal on GitHub (May 22, 2025).
Original GitHub issue: https://github.com/hibiken/asynq/issues/1051

Originally assigned to: @hibiken, @kamikazechaser on GitHub.

Describe the bug
Tasks are accumulating over time in the active queue and never get processed. It seems like they accumulate in batches. Last time an increase of stale tasks happened (>1000), it happened during Redis failure under high load. We had a problem, which resulted in 40k tasks failing in a matter of seconds, redis started to return TOO_MANY_REQUESTS and other errors.

After that, the number of active tasks in queue increased forever (It is not the first time this happens).
The stuck tasks are completed in reality, just not deleted from the active queue. Also, they do not have corresponding entries in lease.

Environment (please complete the following information):

  • OS: Linux
  • asynq v0.25.1
  • Redis

To Reproduce
Not sure how to reproduce this behavior. Maybe stress redis until it starts returning TOO_MANY_REQUESTS might help, because this are the conditions under which it happened for us.

Screenshots
Image
Image
Image

Originally created by @asadzeynal on GitHub (May 22, 2025). Original GitHub issue: https://github.com/hibiken/asynq/issues/1051 Originally assigned to: @hibiken, @kamikazechaser on GitHub. **Describe the bug** Tasks are accumulating over time in the active queue and never get processed. It seems like they accumulate in batches. Last time an increase of stale tasks happened (>1000), it happened during Redis failure under high load. We had a problem, which resulted in 40k tasks failing in a matter of seconds, redis started to return TOO_MANY_REQUESTS and other errors. After that, the number of active tasks in queue increased forever (It is not the first time this happens). The stuck tasks are completed in reality, just not deleted from the active queue. Also, they do not have corresponding entries in **lease**. **Environment (please complete the following information):** - OS: Linux - asynq v0.25.1 - Redis **To Reproduce** Not sure how to reproduce this behavior. Maybe stress redis until it starts returning TOO_MANY_REQUESTS might help, because this are the conditions under which it happened for us. **Screenshots** ![Image](https://github.com/user-attachments/assets/0bdca113-99b1-4af5-9aac-ee5b320bc40f) ![Image](https://github.com/user-attachments/assets/b51cc62a-db93-4ff8-af03-52e0b11108f4) <img width="1300" alt="Image" src="https://github.com/user-attachments/assets/f9eee7eb-3cf7-40d7-b93a-9e2a39fc18b8" />
kerem commented 2026-03-02 05:37:55 +03:00
Author
Owner
Copy link

@kamikazechaser commented on GitHub (May 22, 2025):

TOO_MANY_REQUESTS

Could you mention where and how redis is hosted?

<!-- gh-comment-id:2901251694 --> @kamikazechaser commented on GitHub (May 22, 2025): > TOO_MANY_REQUESTS Could you mention where and how redis is hosted?
kerem commented 2026-03-02 05:37:55 +03:00
Author
Owner
Copy link

@Mdhesari commented on GitHub (Aug 18, 2025):

@asadzeynal please check this out and help me complete:
https://github.com/Mdhesari/asynq-issue-reproduction

here's the similar repduction of your issue, please take a look, I have used ai for faster reproduction after different kinds of prompts...

<!-- gh-comment-id:3197638808 --> @Mdhesari commented on GitHub (Aug 18, 2025): @asadzeynal please check this out and help me complete: https://github.com/Mdhesari/asynq-issue-reproduction here's the similar repduction of your issue, please take a look, I have used ai for faster reproduction after different kinds of prompts...
kerem commented 2026-03-02 05:37:55 +03:00
Author
Owner
Copy link

@Ning-Qing commented on GitHub (Dec 26, 2025):

I've also encountered the same problem. May I ask if there are any solutions

<!-- gh-comment-id:3692396495 --> @Ning-Qing commented on GitHub (Dec 26, 2025): I've also encountered the same problem. May I ask if there are any solutions
kerem commented 2026-03-02 05:37:55 +03:00
Author
Owner
Copy link

@Ning-Qing commented on GitHub (Dec 26, 2025):

@kamikazechaser help

<!-- gh-comment-id:3692430899 --> @Ning-Qing commented on GitHub (Dec 26, 2025): @kamikazechaser help
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
dependabot/github_actions/golangci/golangci-lint-action-9
dependabot/github_actions/codecov/codecov-action-6
dependabot/github_actions/actions/download-artifact-8
dependabot/github_actions/actions/upload-artifact-7
dependabot/github_actions/actions/checkout-6
dependabot/go_modules/tools/github.com/hibiken/asynq-0.26.0
dependabot/go_modules/tools/github.com/redis/go-redis/v9-9.18.0
dependabot/go_modules/github.com/redis/go-redis/v9-9.18.0
dependabot/go_modules/x/github.com/hibiken/asynq-0.26.0
dependabot/go_modules/tools/github.com/mattn/go-runewidth-0.0.23
dependabot/go_modules/x/github.com/redis/go-redis/v9-9.18.0
dependabot/go_modules/golang.org/x/sys-0.43.0
dependabot/go_modules/tools/github.com/prometheus/client_golang-1.23.2
dependabot/go_modules/google.golang.org/protobuf-1.36.11
dependabot/go_modules/tools/github.com/spf13/pflag-1.0.10
dependabot/go_modules/golang.org/x/time-0.15.0
dependabot/go_modules/x/github.com/prometheus/client_golang-1.23.2
sohail/v0.27.0
pr-1100
next
preview-v0.26.0
sohail/recoverer-fix
sohail/unix-stop-signal
sohail/pm-redis-conn-hotfix
sohail/release-v0.25.0
revert-937-dependabot/go_modules/golang.org/x/time-0.7.0
develop
sohail/changelog/v0.25.0
go-versions
sohail/activate-dependabot
revert-491-master
v0.26.0
v0.25.1
v0.25.0
v0.24.1
v0.24.0
v0.23.0
v0.22.1
v0.22.0
v0.21.0
v0.20.0
v0.19.1
v0.19.0
v0.18.6
v0.18.5
v0.18.4
v0.18.3
v0.18.2
v0.18.1
v0.18.0
v0.17.2
v0.17.1
v0.17.0
v0.16.1
v0.16.0
v0.15.0
v0.14.1
v0.14.0
v0.13.1
v0.13.0
v0.12.0
v0.11.0
v0.10.0
v0.10.0.rc1
v0.9.4
v0.9.3
v0.9.2
v0.9.1
v0.9.0
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.1
v0.7.0
v0.6.2
v0.6.1
v0.6.0
v0.5.0
v0.4.0
v0.3.0
v0.2.2
v0.2.1
v0.2.0
v0.1.0
Labels
Clear labels   
CLI

   
bug

   
designing

   
documentation

   
duplicate

   
enhancement

   
good first issue

   
good first issue

   
help wanted

   
idea

   
invalid

   
investigate

   
needs-more-info

   
performance

   
pr-welcome

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
wontfix

   
work in progress

   
work in progress

   
work-around-available
No labels
CLI
bug
designing
documentation
duplicate
enhancement
good first issue
good first issue
help wanted
idea
invalid
investigate
needs-more-info
performance
pr-welcome
pull-request
question
wontfix
work in progress
work in progress
work-around-available
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/asynq#502
No description provided.