[GH-ISSUE #1089] [BUG] The heartbeat does not actually check whether a worker is truly alive when renewing its lease. #2546

Open
opened 2026-03-15 20:49:57 +03:00 by kerem · 0 comments
Owner

Originally created by @lgYanami on GitHub (Jan 4, 2026).
Original GitHub issue: https://github.com/hibiken/asynq/issues/1089

Originally assigned to: @hibiken, @kamikazechaser on GitHub.

Describe the bug
There appears to be a bug in the system.

The heartbeat does not actually check whether a worker is truly alive when renewing its lease. If a worker crashes and fails to notify the heartbeat via a "finished" channel, the heartbeat will continue to renew the lease for this crashed worker. The lease will only expire when the worker's deadline is reached, triggering retry logic. However, until that point, the lease remains active and is only replaced when a new lease takes over. Before the deadline is reached, this zombie worker will continue to occupy a concurrency slot.

Originally created by @lgYanami on GitHub (Jan 4, 2026). Original GitHub issue: https://github.com/hibiken/asynq/issues/1089 Originally assigned to: @hibiken, @kamikazechaser on GitHub. **Describe the bug** There appears to be a bug in the system. The heartbeat does not actually check whether a worker is truly alive when renewing its lease. If a worker crashes and fails to notify the heartbeat via a "finished" channel, the heartbeat will continue to renew the lease for this crashed worker. The lease will only expire when the worker's deadline is reached, triggering retry logic. However, until that point, the lease remains active and is only replaced when a new lease takes over. Before the deadline is reached, this zombie worker will continue to occupy a concurrency slot.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/asynq#2546
No description provided.