starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00

[GH-ISSUE #525] [BUG] The task cannot be resumed after the worker is shut down while the task is running #245

New issue

Open

opened 2026-03-02 05:19:54 +03:00 by kerem · 2 comments

kerem commented

2026-03-02 05:19:54 +03:00

Owner

Originally created by @zijiwork on GitHub (Aug 3, 2022).
Original GitHub issue: https://github.com/hibiken/asynq/issues/525

Originally assigned to: @hibiken on GitHub.

Describe the bug
When the task is running, I try to shut down the worker, but restarting the worker task cannot recover normally, and the status of the Asynq Monitoring task is active. and cannot be closed

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):

go tasks.TaskWorker()

TaskWorker

	// start server
	if err := srv.Start(mux); err != nil {
		common.LOG.Error(fmt.Sprintf("could not start server: %v", err))
	}

	quit := make(chan os.Signal, 1)
	signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
	<-quit // wait for termination signal
	log.Println("[asynq] Receive exit signal task worker exit")
	srv.Shutdown()

try restarting the asynq worker

2022/08/03 - 14:29:31.445       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:31 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
2022/08/03 - 14:29:34.577       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:34 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
2022/08/03 - 14:29:37.699       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:37 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
2022/08/03 - 14:29:40.803       INFO    /Users/micheng/code/luban/controller/application/deploy.go:297  2022-08-03 14:29:40 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting...
[GIN] 2022/08/03 - 14:29:41 | 200 |   26.725875ms |             ::1 | GET      "/task/monitor/api/queues/critical/completed_tasks?page=1&size=100"
Receive SIGTERM ,exiting gracefully....
2022/08/03 14:29:43 [asynq] Receive exit signal task scheduler exit
2022/08/03 14:29:43 [asynq] Receive exit signal task worker exit
asynq: pid=49878 2022/08/03 06:29:43.051790 INFO: Starting graceful shutdown
asynq: pid=49878 2022/08/03 06:29:43.051778 INFO: Scheduler shutting down
2022/08/03 14:29:43 [agent] Receive exit signal web server exit
2022/08/03 14:29:43 Shutting down server...
2022/08/03 14:29:43 [agent] Receive exit signal rpc server exit

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

Environment (please complete the following information):

OS: MacOS
Version asynq v0.19.0 asynqmon v0.4.0

Originally created by @zijiwork on GitHub (Aug 3, 2022). Original GitHub issue: https://github.com/hibiken/asynq/issues/525 Originally assigned to: @hibiken on GitHub. **Describe the bug** When the task is running, I try to shut down the worker, but restarting the worker task cannot recover normally, and the status of the Asynq Monitoring task is active. and cannot be closed **To Reproduce** Steps to reproduce the behavior (Code snippets if applicable): ``` go tasks.TaskWorker() ``` ## TaskWorker ``` // start server if err := srv.Start(mux); err != nil { common.LOG.Error(fmt.Sprintf("could not start server: %v", err)) } quit := make(chan os.Signal, 1) signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT) <-quit // wait for termination signal log.Println("[asynq] Receive exit signal task worker exit") srv.Shutdown() ``` ## try restarting the asynq worker ``` 2022/08/03 - 14:29:31.445 INFO /Users/micheng/code/luban/controller/application/deploy.go:297 2022-08-03 14:29:31 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting... 2022/08/03 - 14:29:34.577 INFO /Users/micheng/code/luban/controller/application/deploy.go:297 2022-08-03 14:29:34 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting... 2022/08/03 - 14:29:37.699 INFO /Users/micheng/code/luban/controller/application/deploy.go:297 2022-08-03 14:29:37 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting... 2022/08/03 - 14:29:40.803 INFO /Users/micheng/code/luban/controller/application/deploy.go:297 2022-08-03 14:29:40 - [task-rpc] - [updated_replicas:1, replicas:2, available_replicas:1, observed_generation:189] waiting... [GIN] 2022/08/03 - 14:29:41 | 200 | 26.725875ms | ::1 | GET "/task/monitor/api/queues/critical/completed_tasks?page=1&size=100" Receive SIGTERM ,exiting gracefully.... 2022/08/03 14:29:43 [asynq] Receive exit signal task scheduler exit 2022/08/03 14:29:43 [asynq] Receive exit signal task worker exit asynq: pid=49878 2022/08/03 06:29:43.051790 INFO: Starting graceful shutdown asynq: pid=49878 2022/08/03 06:29:43.051778 INFO: Scheduler shutting down 2022/08/03 14:29:43 [agent] Receive exit signal web server exit 2022/08/03 14:29:43 Shutting down server... 2022/08/03 14:29:43 [agent] Receive exit signal rpc server exit ``` **Expected behavior** A clear and concise description of what you expected to happen. **Screenshots** ![image](https://user-images.githubusercontent.com/24753365/182541889-abaa3620-cd41-43fc-865d-17e8f7410fe5.png) **Environment (please complete the following information):** - OS: MacOS - Version `asynq v0.19.0` `asynqmon v0.4.0`

kerem added the

bug

label

2026-03-02 05:19:54 +03:00

kerem commented

2026-03-02 05:19:55 +03:00

Author

Owner

@dxl3811051 commented on GitHub (Aug 5, 2022):

Hello, I want to ask you how to stop a running task. I click the stop button, but the task is still running. My task is a simple loop print

@dxl3811051 commented on GitHub (Aug 5, 2022): Hello, I want to ask you how to stop a running task. I click the stop button, but the task is still running. My task is a simple loop print

kerem commented

2026-03-02 05:19:55 +03:00

Author

Owner

@liran commented on GitHub (Jun 27, 2023):

I've also had situations where tasks get stuck in active state and can't be resumed.
asynq version 0.24.1, redis 7.0.7 cluster

These 3 tasks are in the active list

But the task state is still pending

@liran commented on GitHub (Jun 27, 2023): I've also had situations where tasks get stuck in active state and can't be resumed. asynq version 0.24.1, redis 7.0.7 cluster <img width="511" alt="image" src="https://github.com/hibiken/asynq/assets/22112008/790b6ad4-c879-4a33-8ce7-e6c4214d68d7"> These 3 tasks are in the `active` list <img width="1293" alt="image" src="https://github.com/hibiken/asynq/assets/22112008/7c57c660-3caa-430f-80e9-e49bacf5f6ea"> But the task `state` is still `pending` <img width="1434" alt="image" src="https://github.com/hibiken/asynq/assets/22112008/4f95c47e-d4a8-42f4-bbc8-d9302f6d2f06">

kerem referenced this issue

2026-03-02 06:05:11 +03:00

[PR #252] [MERGED] Refactor redis key and store messages in protobuf #688

kerem referenced this issue

2026-03-02 06:05:14 +03:00

[PR #287] [MERGED] v0.18 #700

kerem referenced this issue

2026-03-07 22:05:45 +03:00

[GH-ISSUE #242] [FEATURE REQUEST] Version 1? #1104

kerem referenced this issue

2026-03-07 22:05:46 +03:00

[GH-ISSUE #245] Use protobuf to encode messages stored in Redis #1106