[PR #174] [MERGED] Automatic recovery of orphan tasks #650

New issue

Closed

opened 2026-03-02 05:54:25 +03:00 by kerem · 0 comments

kerem commented

2026-03-02 05:54:25 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hibiken/asynq/pull/174
Author: @hibiken
Created: 6/22/2020
Status: ✅ Merged
Merged: 6/29/2020
Merged by: @hibiken

Base: next ← Head: enhance/fault-tolerance

📝 Commits (10+)

825b29b Add redis key for deadlines in base package
7fddba8 Change TaskMessage Timeout and Deadline to int
9d471e3 Use default timeout of 30mins if both timeout and deadline are not
5587601 Add task message to deadlines set on dequeue
d98fc7c Update RDB.Dequeue to return message and deadline
cf3ccdf Update RDB.Done to remove message from deadlines set
ee31ea2 Update RDB.Retry to remove message from deadlines set
c1a3584 Update RDB.Kill to remove message from deadlines set
d334a66 Update RDB.Dequeue to return deadline as time.Time
889e801 Update RDB.Requeue to remove message from deadlines set

📊 Changes

20 files changed (+1348 additions, -358 deletions)

View changed files

📝 CHANGELOG.md (+7 -0)
📝 client.go (+41 -5)
📝 client_test.go (+48 -33)
📝 context.go (+3 -14)
📝 context_test.go (+40 -49)
📝 internal/asynqtest/asynqtest.go (+33 -5)
📝 internal/base/base.go (+14 -21)
📝 internal/base/base_test.go (+16 -19)
📝 internal/rdb/rdb.go (+132 -56)
📝 internal/rdb/rdb_test.go (+399 -85)
📝 internal/testbroker/testbroker.go (+14 -2)
📝 processor.go (+66 -48)
📝 processor_test.go (+4 -17)
➕ recoverer.go (+96 -0)
➕ recoverer_test.go (+162 -0)
📝 server.go (+10 -0)
📝 syncer.go (+6 -2)
📝 syncer_test.go (+42 -2)
➕ tools/asynq/cmd/migrate.go (+214 -0)
📝 tools/go.mod (+1 -0)

📄 Description

Currently when server crashes while processing tasks, the tasks get stuck in in-progress queue forever.

This PR introduces recovery mechanism for such cases.

Changes:

All tasks will have deadline (specified via Timeout or Deadline option, by default if none are specified it'll use 30 mins timeout)
Tasks which exceeded its deadline will be retried, and treated as failed.

TODO:

Document default 30mins timeout
Changelog
Expose migrate command to migrate all tasks enqueued with previous version to smooth the version upgrade

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hibiken/asynq/pull/174 **Author:** [@hibiken](https://github.com/hibiken) **Created:** 6/22/2020 **Status:** ✅ Merged **Merged:** 6/29/2020 **Merged by:** [@hibiken](https://github.com/hibiken) **Base:** `next` ← **Head:** `enhance/fault-tolerance` --- ### 📝 Commits (10+) - [`825b29b`](https://github.com/hibiken/asynq/commit/825b29b7f3b9e50fe0e1a6b67f545eb905f6ad2c) Add redis key for deadlines in base package - [`7fddba8`](https://github.com/hibiken/asynq/commit/7fddba8f10f5c0e6016192100e983f4b7b25be45) Change TaskMessage Timeout and Deadline to int - [`9d471e3`](https://github.com/hibiken/asynq/commit/9d471e3aa260728f0b2e5630184818f075ec816c) Use default timeout of 30mins if both timeout and deadline are not - [`5587601`](https://github.com/hibiken/asynq/commit/5587601b815c3b1774b05b774393ad6516e9ead9) Add task message to deadlines set on dequeue - [`d98fc7c`](https://github.com/hibiken/asynq/commit/d98fc7c419edc3a4377f90a2f07164cc087abb10) Update RDB.Dequeue to return message and deadline - [`cf3ccdf`](https://github.com/hibiken/asynq/commit/cf3ccdfa920773f3718e6689e5f3314a7624fb11) Update RDB.Done to remove message from deadlines set - [`ee31ea2`](https://github.com/hibiken/asynq/commit/ee31ea2e9c5230af8c312cdbb0ecbf5c91998f00) Update RDB.Retry to remove message from deadlines set - [`c1a3584`](https://github.com/hibiken/asynq/commit/c1a358418aee278a7aacc817667bfed10244e9b9) Update RDB.Kill to remove message from deadlines set - [`d334a66`](https://github.com/hibiken/asynq/commit/d334a664b81f3e6d3691cf0d8bf5ad3e5ea2df0f) Update RDB.Dequeue to return deadline as time.Time - [`889e801`](https://github.com/hibiken/asynq/commit/889e8010f42523ae95af1f0e9bf4e86e164bbd0e) Update RDB.Requeue to remove message from deadlines set ### 📊 Changes **20 files changed** (+1348 additions, -358 deletions) <details> <summary>View changed files</summary> 📝 `CHANGELOG.md` (+7 -0) 📝 `client.go` (+41 -5) 📝 `client_test.go` (+48 -33) 📝 `context.go` (+3 -14) 📝 `context_test.go` (+40 -49) 📝 `internal/asynqtest/asynqtest.go` (+33 -5) 📝 `internal/base/base.go` (+14 -21) 📝 `internal/base/base_test.go` (+16 -19) 📝 `internal/rdb/rdb.go` (+132 -56) 📝 `internal/rdb/rdb_test.go` (+399 -85) 📝 `internal/testbroker/testbroker.go` (+14 -2) 📝 `processor.go` (+66 -48) 📝 `processor_test.go` (+4 -17) ➕ `recoverer.go` (+96 -0) ➕ `recoverer_test.go` (+162 -0) 📝 `server.go` (+10 -0) 📝 `syncer.go` (+6 -2) 📝 `syncer_test.go` (+42 -2) ➕ `tools/asynq/cmd/migrate.go` (+214 -0) 📝 `tools/go.mod` (+1 -0) </details> ### 📄 Description Currently when server crashes while processing tasks, the tasks get stuck in in-progress queue forever. This PR introduces recovery mechanism for such cases. Changes: - All tasks will have deadline (specified via Timeout or Deadline option, by default if none are specified it'll use 30 mins timeout) - Tasks which exceeded its deadline will be retried, and treated as failed. TODO: - [x] Document default 30mins timeout - [x] Changelog - [x] Expose `migrate` command to migrate all tasks enqueued with previous version to smooth the version upgrade --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>