[GH-ISSUE #420] [BUG] Task stuck in active state forever #1200

New issue

Open

opened 2026-03-07 22:07:17 +03:00 by kerem · 15 comments

kerem commented 2026-03-07 22:07:17 +03:00

Owner

Copy link

Originally created by @mailbaoer on GitHub (Mar 17, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/420

Originally assigned to: @hibiken on GitHub.

Describe the bug
I have some tasks not set timeout, for a long time run, I found some tasks are always in running state, I tried to cancel them in web ui or cli, they can't be canceled, and the state changed to running after canceling

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):
sorry, I don't know how reproduce

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Environment (please complete the following information):

• OS: [e.g. MacOS, Linux]
• Version of asynq package [e.g. v1.0.0]

Additional context
Add any other context about the problem here.

Originally created by @mailbaoer on GitHub (Mar 17, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/420 Originally assigned to: @hibiken on GitHub. **Describe the bug** I have some tasks not set timeout, for a long time run, I found some tasks are always in running state, I tried to cancel them in web ui or cli, they can't be canceled, and the state changed to running after canceling **To Reproduce** Steps to reproduce the behavior (Code snippets if applicable): sorry, I don't know how reproduce **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots**  **Environment (please complete the following information):** - OS: [e.g. MacOS, Linux] - Version of `asynq` package [e.g. v1.0.0] **Additional context** Add any other context about the problem here.

kerem added the

bug

label 2026-03-07 22:07:17 +03:00

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@hibiken commented on GitHub (Mar 17, 2022):

@mailbaoer Thank you for opening an issue!

Would you mind providing the version of asynq package you are using :)

<!-- gh-comment-id:1070089911 --> @hibiken commented on GitHub (Mar 17, 2022): @mailbaoer Thank you for opening an issue! Would you mind providing the version of asynq package you are using :)

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@mailbaoer commented on GitHub (Mar 17, 2022):

I'm use 0.22.1 now, but this bug may exists before this version, I've see this in other versions, maybe 0.18 for my first time use asynq

<!-- gh-comment-id:1070543649 --> @mailbaoer commented on GitHub (Mar 17, 2022): I'm use 0.22.1 now, but this bug may exists before this version, I've see this in other versions, maybe 0.18 for my first time use asynq

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@hibiken commented on GitHub (Mar 17, 2022):

I see.

We've made some improvements around orphaned task recovery in v0.22. If you are using latest version of Web UI (v0.6.0), you'll see the status of the tasks will show "Orphaned". This happens when a worker start working on a task but crashes before completing the processing.

If you run a server against the same queue, they'll be recovered automatically after some time period (i.e. after a few heartbeat misses). Once the task is orphaned, they are no longer cancelable (the latest web UI will disable the cancel button)

Follow up questions:

• Are you running a server against the queue these tasks are in?
• Would you mind running this redis command to see what's in the lease set? (ZRANGE asynq:{default}:lease 0 -1 WITHSCORES)

<!-- gh-comment-id:1070888928 --> @hibiken commented on GitHub (Mar 17, 2022): I see. We've made some improvements around orphaned task recovery in v0.22. If you are using latest version of Web UI (v0.6.0), you'll see the status of the tasks will show "Orphaned". This happens when a worker start working on a task but crashes before completing the processing. If you run a server against the same queue, they'll be recovered automatically after some time period (i.e. after a few heartbeat misses). Once the task is orphaned, they are no longer cancelable (the latest web UI will disable the cancel button) Follow up questions: - Are you running a server against the queue these tasks are in? - Would you mind running this redis command to see what's in the lease set? (`ZRANGE asynq:{default}:lease 0 -1 WITHSCORES`)

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@mailbaoer commented on GitHub (Mar 18, 2022):

1. Yes, I started a worker service on my server only run tasks for background, but NewxxxTask run with another api service to add tasks
2. I've run this command,but does not see anything
   

<!-- gh-comment-id:1071969094 --> @mailbaoer commented on GitHub (Mar 18, 2022): 1. Yes, I started a worker service on my server only run tasks for background, but NewxxxTask run with another api service to add tasks 2. I've run this command,but does not see anything 

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@hibiken commented on GitHub (Mar 18, 2022):

Ok, thanks for providing that info.

Would you mind running this command:

ZRANGE asynq:{default}:deadlines 0 -1 WITHSCORES

<!-- gh-comment-id:1071995010 --> @hibiken commented on GitHub (Mar 18, 2022): Ok, thanks for providing that info. Would you mind running this command: ``` ZRANGE asynq:{default}:deadlines 0 -1 WITHSCORES ```

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@mailbaoer commented on GitHub (Mar 18, 2022):

all the tasks stays in running for a lot of days, It's still running now, and I'm upgrade asynqmon to 0.6.1

<!-- gh-comment-id:1072009750 --> @mailbaoer commented on GitHub (Mar 18, 2022):  all the tasks stays in running for a lot of days, It's still running now, and I'm upgrade asynqmon to 0.6.1 

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@hibiken commented on GitHub (Mar 18, 2022):

That's very strange. I thought you'd have entries in either asynq:{default}:deadlines (used by v0.21.x or below) or asynq:{default}:lease (used by v0.22.x). These zsets are used to recover orphaned tasks in case of worker crash, but the fact that there's no entries there seem something unexpected happened.
I'll keep this bug open to see if others have encountered similar issue and get more context.

Please let me know if you can reproduce this, I'd like to know how to reproduce this bug.

---

If you need to address this manually, you can get a list of "active" tasks and put their IDs back in the pending list (note: the IDs you see in the image above is just a prefix, so make sure to click into each row to get the full ID)
Once you have the IDs, you can

1. delete them from the active list (LREM asynq:{default}:active 1 <task_id>
2. put them back on the pending list LPUSH asynq:{default}:pending <task_id>

<!-- gh-comment-id:1072384232 --> @hibiken commented on GitHub (Mar 18, 2022): That's very strange. I thought you'd have entries in either `asynq:{default}:deadlines` (used by v0.21.x or below) or `asynq:{default}:lease` (used by v0.22.x). These zsets are used to recover orphaned tasks in case of worker crash, but the fact that there's no entries there seem something unexpected happened. I'll keep this bug open to see if others have encountered similar issue and get more context. Please let me know if you can reproduce this, I'd like to know how to reproduce this bug. --- If you need to address this manually, you can get a list of "active" tasks and put their IDs back in the pending list (note: the IDs you see in the image above is just a prefix, so make sure to click into each row to get the full ID) Once you have the IDs, you can 1) delete them from the active list (`LREM asynq:{default}:active 1 <task_id>` 2) put them back on the pending list `LPUSH asynq:{default}:pending <task_id>`

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@mailbaoer commented on GitHub (Mar 21, 2022):

Thank you very much for your patience in answering, if I encounter this problem again I will check to see if I can reproduce it

<!-- gh-comment-id:1073429748 --> @mailbaoer commented on GitHub (Mar 21, 2022): Thank you very much for your patience in answering, if I encounter this problem again I will check to see if I can reproduce it

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@namhq1989 commented on GitHub (Mar 22, 2022):

any update? I met this bug too. asynq v0.22.1, redis v5.0.7

<!-- gh-comment-id:1074847629 --> @namhq1989 commented on GitHub (Mar 22, 2022): any update? I met this bug too. asynq v0.22.1, redis v5.0.7

kerem commented 2026-03-07 22:07:18 +03:00

Author

Owner

Copy link

@hibiken commented on GitHub (Mar 22, 2022):

@namhq1989 Thanks for the comment. We're looking for a way to reproduce this.

For anyone experienced this bug, please provide the following:

• If possible, the steps to reproduce this bug.
• Asynq version
• Output of the following command (replace the <qname> with the queue name (e.g. asynq:{default}:lease):
  • (for asynq v0.22.x or above): ZRANGE asynq:{<qname>}:lease 0 -1 WITHSCORES
  • (for asynq v0.21.x or below): ZRANGE asynq:{<qname>}:deadlines 0 -1 WITHSCORES
• Whether the IDs of the orphaned tasks are in the output above

<!-- gh-comment-id:1075146735 --> @hibiken commented on GitHub (Mar 22, 2022): @namhq1989 Thanks for the comment. We're looking for a way to reproduce this. For anyone experienced this bug, please provide the following: - If possible, the steps to reproduce this bug. - Asynq version - Output of the following command (replace the `<qname>` with the queue name (e.g. `asynq:{default}:lease`): - (for asynq v0.22.x or above): `ZRANGE asynq:{<qname>}:lease 0 -1 WITHSCORES` - (for asynq v0.21.x or below): `ZRANGE asynq:{<qname>}:deadlines 0 -1 WITHSCORES` - Whether the IDs of the orphaned tasks are in the output above

kerem commented 2026-03-07 22:07:19 +03:00

Author

Owner

Copy link

@dokudoki commented on GitHub (Mar 31, 2022):

• I think I ran into this issue when the worker throw a "fatal error" and exited.
• Asynq version v0.21.0

 1) "4d841b6c-1de5-4a70-9233-db16a2493831"
 2) "1648764430"
 3) "af985397-6ee3-4f4d-825d-e926a0a6d1cd"
 4) "1648764430"
 5) "10dff61b-9f5c-4d30-a2f6-42cdb9a05f91"
 6) "1648764440"
 7) "45a12b75-0b83-4e2b-bc7c-93357207ed36"
 8) "1648764440"
 9) "d40431d2-e634-4922-acd0-ab435ec6ac47"
10) "1648764440"
11) "2117339c-db7a-4790-a255-7c9ef0a19b74"
12) "1648765083"
13) "5c1fb99c-60ec-49b1-b7d1-04d953c9d9f0"
14) "1648765083"
15) "932ad61e-d2c1-4f7f-8719-9af9db04f37a"
16) "1648765882"
17) "e1ae65d7-978c-4ac0-a152-3312e458df5e"
18) "1648765882"

<!-- gh-comment-id:1085173782 --> @dokudoki commented on GitHub (Mar 31, 2022): - I think I ran into this issue when the worker throw a "fatal error" and exited. - Asynq version v0.21.0 ``` 1) "4d841b6c-1de5-4a70-9233-db16a2493831" 2) "1648764430" 3) "af985397-6ee3-4f4d-825d-e926a0a6d1cd" 4) "1648764430" 5) "10dff61b-9f5c-4d30-a2f6-42cdb9a05f91" 6) "1648764440" 7) "45a12b75-0b83-4e2b-bc7c-93357207ed36" 8) "1648764440" 9) "d40431d2-e634-4922-acd0-ab435ec6ac47" 10) "1648764440" 11) "2117339c-db7a-4790-a255-7c9ef0a19b74" 12) "1648765083" 13) "5c1fb99c-60ec-49b1-b7d1-04d953c9d9f0" 14) "1648765083" 15) "932ad61e-d2c1-4f7f-8719-9af9db04f37a" 16) "1648765882" 17) "e1ae65d7-978c-4ac0-a152-3312e458df5e" 18) "1648765882" ```

kerem commented 2026-03-07 22:07:19 +03:00

Author

Owner

Copy link

@piperck commented on GitHub (May 4, 2022):

I met this too.. If it happens again, I will try to investigate it.

<!-- gh-comment-id:1117378371 --> @piperck commented on GitHub (May 4, 2022): I met this too.. If it happens again, I will try to investigate it.

kerem commented 2026-03-07 22:07:19 +03:00

Author

Owner

Copy link

@paveljanda commented on GitHub (May 12, 2022):

Hi, we never had this problem as we never used asynq before. BUT if anyone could describe how to reproduce this bug and it gets fixed/solved, it would help us choosing the right distributed task queue for our projects. Currently choosing from ~5 contestants.

Thanks a lot to everyone!

<!-- gh-comment-id:1124756284 --> @paveljanda commented on GitHub (May 12, 2022): Hi, we never had this problem as we never used asynq before. BUT if anyone could describe how to reproduce this bug and it gets fixed/solved, it would help us choosing the right distributed task queue for our projects. Currently choosing from ~5 contestants. Thanks a lot to everyone!

kerem commented 2026-03-07 22:07:19 +03:00

Author

Owner

Copy link

@zijiwork commented on GitHub (Jul 15, 2022):

I encountered this problem in the following versions, the task is always in the active state after the worker restarts, and the task cannot be closed

asynq v0.19.0
asynqmon v0.4.0
redis_version 5.0.4

luban:0>ZRANGE asynq:{critical}:deadlines 0 -1 WITHSCORES
1) "bab56af5-f43e-4192-a5f3-b7eb03246b8b"
2) "1657874411"
3) "f559737f-f729-4716-bfc5-ed0fe6a720a4"
4) "1657874530"
5) "3b839bf5-9ba8-43c9-bc21-906a925e945e"
6) "1657875931"
7) "77147b6f-095d-47d6-a2ce-35d3ad3d80e2"
8) "1657875957"

<!-- gh-comment-id:1185309282 --> @zijiwork commented on GitHub (Jul 15, 2022): I encountered this problem in the following versions, the task is always in the active state after the worker restarts, and the task cannot be closed ``` asynq v0.19.0 asynqmon v0.4.0 redis_version 5.0.4 ``` ``` luban:0>ZRANGE asynq:{critical}:deadlines 0 -1 WITHSCORES 1) "bab56af5-f43e-4192-a5f3-b7eb03246b8b" 2) "1657874411" 3) "f559737f-f729-4716-bfc5-ed0fe6a720a4" 4) "1657874530" 5) "3b839bf5-9ba8-43c9-bc21-906a925e945e" 6) "1657875931" 7) "77147b6f-095d-47d6-a2ce-35d3ad3d80e2" 8) "1657875957" ```

kerem commented 2026-03-07 22:07:19 +03:00

Author

Owner

Copy link

@KrokoYR commented on GitHub (Oct 8, 2024):

We accidentally met this kind of issue. What we did:

• deployed a new queue service(let's call it NEW) with such setup: redis.Options{Addr: "{addr}", DB: 10}
• already existing service(let's call it OLD) has the following setup: redis.Options{Addr: "{addr}", DB: 0}
• both of each services have queue with same name, let's say some_queue

This led to a problem that those services started "stealing" each others tasks. So we made a mistake, that we didn't pass DB number into asynq.RedisClientOpt. Maybe it will help someone

<!-- gh-comment-id:2400412514 --> @KrokoYR commented on GitHub (Oct 8, 2024): We accidentally met this kind of issue. What we did: - deployed a new queue service(let's call it NEW) with such setup: `redis.Options{Addr: "{addr}", DB: 10}` - already existing service(let's call it OLD) has the following setup: `redis.Options{Addr: "{addr}", DB: 0}` - both of each services have queue with same name, let's say `some_queue` This led to a problem that those services started "stealing" each others tasks. So we made a mistake, that we didn't pass DB number into `asynq.RedisClientOpt`. Maybe it will help someone

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

master

dependabot/github_actions/golangci/golangci-lint-action-9

dependabot/github_actions/codecov/codecov-action-6

dependabot/github_actions/actions/download-artifact-8

dependabot/github_actions/actions/upload-artifact-7

dependabot/github_actions/actions/checkout-6

dependabot/go_modules/tools/github.com/hibiken/asynq-0.26.0

dependabot/go_modules/tools/github.com/redis/go-redis/v9-9.18.0

dependabot/go_modules/github.com/redis/go-redis/v9-9.18.0

dependabot/go_modules/x/github.com/hibiken/asynq-0.26.0

dependabot/go_modules/tools/github.com/mattn/go-runewidth-0.0.23

dependabot/go_modules/x/github.com/redis/go-redis/v9-9.18.0

dependabot/go_modules/golang.org/x/sys-0.43.0

dependabot/go_modules/tools/github.com/prometheus/client_golang-1.23.2

dependabot/go_modules/google.golang.org/protobuf-1.36.11

dependabot/go_modules/tools/github.com/spf13/pflag-1.0.10

dependabot/go_modules/golang.org/x/time-0.15.0

dependabot/go_modules/x/github.com/prometheus/client_golang-1.23.2

sohail/v0.27.0

pr-1100

next

preview-v0.26.0

sohail/recoverer-fix

sohail/unix-stop-signal

sohail/pm-redis-conn-hotfix

sohail/release-v0.25.0

revert-937-dependabot/go_modules/golang.org/x/time-0.7.0

develop

sohail/changelog/v0.25.0

go-versions

sohail/activate-dependabot

revert-491-master

v0.26.0

v0.25.1

v0.25.0

v0.24.1

v0.24.0

v0.23.0

v0.22.1

v0.22.0

v0.21.0

v0.20.0

v0.19.1

v0.19.0

v0.18.6

v0.18.5

v0.18.4

v0.18.3

v0.18.2

v0.18.1

v0.18.0

v0.17.2

v0.17.1

v0.17.0

v0.16.1

v0.16.0

v0.15.0

v0.14.1

v0.14.0

v0.13.1

v0.13.0

v0.12.0

v0.11.0

v0.10.0

v0.10.0.rc1

v0.9.4

v0.9.3

v0.9.2

v0.9.1

v0.9.0

v0.8.3

v0.8.2

v0.8.1

v0.8.0

v0.7.1

v0.7.0

v0.6.2

v0.6.1

v0.6.0

v0.5.0

v0.4.0

v0.3.0

v0.2.2

v0.2.1

v0.2.0

v0.1.0

Labels

Clear labels   

CLI

  

bug

  

designing

  

documentation

  

duplicate

  

enhancement

  

good first issue

  

good first issue

  

help wanted

  

idea

  

invalid

  

investigate

  

needs-more-info

  

performance

  

pr-welcome

  

pull-request

Mirrored from GitHub Pull Request

  

question

  

wontfix

  

work in progress

  

work in progress

  

work-around-available

No labels

CLI

bug

designing

documentation

duplicate

enhancement

good first issue

good first issue

help wanted

idea

invalid

investigate

needs-more-info

performance

pr-welcome

pull-request

question

wontfix

work in progress

work in progress

work-around-available

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#1200

No description provided.