[GH-ISSUE #612] [BUG] asynq: task lease expired ，What does it mean？

kerem commented

2026-03-02 05:20:17 +03:00

Owner

Originally created by @XTeam-Wing on GitHub (Feb 11, 2023).
Original GitHub issue: https://github.com/hibiken/asynq/issues/612

Originally assigned to: @hibiken on GitHub.

What is the reason for this error? If the queue is wrong, no error information is seen.

Originally created by @XTeam-Wing on GitHub (Feb 11, 2023). Original GitHub issue: https://github.com/hibiken/asynq/issues/612 Originally assigned to: @hibiken on GitHub. What is the reason for this error? If the queue is wrong, no error information is seen. <img width="1364" alt="image" src="https://user-images.githubusercontent.com/25416365/218240024-ec9b7c41-f05a-4fc7-b7e5-4e22aa52e256.png">

kerem

2026-03-02 05:20:17 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@linhbkhn95 commented on GitHub (Feb 14, 2023):

hi @XTeam-Wing
I think your code or infra will catch some problems

// ErrLeaseExpired error indicates that the task failed because the worker working on the task
// could not extend its lease due to missing heartbeats. The worker may have crashed or got cut off from the network.

I see the queue does not exist error in the image above => no worker processed this task => expired time for processing.
Im not sure.
Feel free to provide your code here to have full overview context.

@linhbkhn95 commented on GitHub (Feb 14, 2023): hi @XTeam-Wing I think your code or infra will catch some problems ``` // ErrLeaseExpired error indicates that the task failed because the worker working on the task // could not extend its lease due to missing heartbeats. The worker may have crashed or got cut off from the network. ``` I see the queue does not exist error in the image above => no worker processed this task => expired time for processing. Im not sure. Feel free to provide your code here to have full overview context.

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@XTeam-Wing commented on GitHub (Feb 24, 2023):

	srv := asynq.NewServer(
		asynq.RedisClientOpt{
			Addr:         conf.Redis.Addr,
			Password:     conf.Redis.Password,
			DB:           conf.Redis.DB,
			ReadTimeout:  -1,
			WriteTimeout: 0,
		},

		asynq.Config{
			Concurrency: concurrent,
			Queues: map[string]int{
				"test":     1,
			},
			StrictPriority: true,
			ErrorHandler:   asynq.ErrorHandlerFunc(reportError),
		},
	)

func handledemo(ctx context.Context, t *asynq.Task) error {

	var d model.Arg
	if err := json.Unmarshal(t.Payload(), &d); err != nil {
		logger.Error(err)
		return err
	}
	
	return nil
}

I think mux.HandleFunc should return a clear error. Users can locate where the error occurred, thank you for your reply.

@XTeam-Wing commented on GitHub (Feb 24, 2023): ``` srv := asynq.NewServer( asynq.RedisClientOpt{ Addr: conf.Redis.Addr, Password: conf.Redis.Password, DB: conf.Redis.DB, ReadTimeout: -1, WriteTimeout: 0, }, asynq.Config{ Concurrency: concurrent, Queues: map[string]int{ "test": 1, }, StrictPriority: true, ErrorHandler: asynq.ErrorHandlerFunc(reportError), }, ) ``` <img width="778" alt="image" src="https://user-images.githubusercontent.com/25416365/221121901-66d94313-9ac9-4421-afa3-6bea6d374ede.png"> ``` func handledemo(ctx context.Context, t *asynq.Task) error { var d model.Arg if err := json.Unmarshal(t.Payload(), &d); err != nil { logger.Error(err) return err } return nil } ``` I think mux.HandleFunc should return a clear error. Users can locate where the error occurred, thank you for your reply.

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@linhbkhn95 commented on GitHub (Feb 27, 2023):

Can you show the enqueued task? We should map the handler with task type correctly.

@linhbkhn95 commented on GitHub (Feb 27, 2023): Can you show the enqueued task? We should map the handler with task type correctly.

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@XTeam-Wing commented on GitHub (Mar 8, 2023):

Can you show the enqueued task? We should map the handler with task type correctly.

Hi, I got a more specific error

x-worker  | asynq: pid=928 2023/03/08 11:15:45.904045 WARN: Lease reset failed for f0686598-bf18-4e67-8263-a6983d5dfc95; lease deadline: 2023-03-08 19:15:32.128419658 +0800 CST m=+165249.263673317

I would like to ask what this means。

@XTeam-Wing commented on GitHub (Mar 8, 2023): > Can you show the enqueued task? We should map the handler with task type correctly. Hi, I got a more specific error ``` x-worker | asynq: pid=928 2023/03/08 11:15:45.904045 WARN: Lease reset failed for f0686598-bf18-4e67-8263-a6983d5dfc95; lease deadline: 2023-03-08 19:15:32.128419658 +0800 CST m=+165249.263673317 ``` <img width="1541" alt="image" src="https://user-images.githubusercontent.com/25416365/223762368-5e3f3c84-9612-40d8-a371-20bf131e3223.png"> I would like to ask what this means。

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@XTeam-Wing commented on GitHub (May 22, 2023):

@linhbkhn95 hi,master.

i think the handler is np.

@XTeam-Wing commented on GitHub (May 22, 2023): @linhbkhn95 hi,master. <img width="756" alt="image" src="https://github.com/hibiken/asynq/assets/25416365/9e4423a6-fff3-4b2f-96c6-ea7c49ec757a"> i think the handler is np.

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@Xenov-X commented on GitHub (Jan 12, 2024):

@XTeam-Wing - Did you find a solution to this?

@Xenov-X commented on GitHub (Jan 12, 2024): @XTeam-Wing - Did you find a solution to this?

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@XTeam-Wing commented on GitHub (Jan 13, 2024):

@XTeam-Wing - Did you find a solution to this?

not yet,but I made some changes and this feature doesn't affect me

@XTeam-Wing commented on GitHub (Jan 13, 2024): > @XTeam-Wing - Did you find a solution to this? not yet,but I made some changes and this feature doesn't affect me

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@william9x commented on GitHub (Apr 3, 2024):

@XTeam-Wing I'm facing the same issues, do you mind sharing what changes did you make? Thank you!

@william9x commented on GitHub (Apr 3, 2024): @XTeam-Wing I'm facing the same issues, do you mind sharing what changes did you make? Thank you!

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@Xenov-X commented on GitHub (Apr 3, 2024):

Just to share what I found when doing further testing, which may help you resolve the issue.

My use case:

Asynq redis is exposed to remote workers via a mutual TLS gateway

My issue seemed to arise from an issue when performing high network throughput tasks on EC2, which triggered some kind of rate limiting on outbound network traffic. The remote workers failed to reach the redis DB to perform their heartbeat for an extended period of time, which resulted in the server recording the tasks as failed with task lease expired.

I guess the point I'm trying to make, it the error is due to heartbeats not happening, therefore the task lease doesnt get extended.

The fix in my case, was to apply rate limiting to the task actions to ensure the AWS rate limit issue wasn't triggered.

I'd look at your network path from the worker to the redis port, and see if theres anything that could be stopping the heatbeat connection occurring (even intermittently)

@Xenov-X commented on GitHub (Apr 3, 2024): Just to share what I found when doing further testing, which may help you resolve the issue. My use case: - Asynq redis is exposed to remote workers via a mutual TLS gateway My issue seemed to arise from an issue when performing high network throughput tasks on EC2, which triggered some kind of rate limiting on outbound network traffic. The remote workers failed to reach the redis DB to perform their heartbeat for an extended period of time, which resulted in the server recording the tasks as failed with `task lease expired`. I guess the point I'm trying to make, it the error is due to heartbeats not happening, therefore the task lease doesnt get extended. The fix in my case, was to apply rate limiting to the task actions to ensure the AWS rate limit issue wasn't triggered. I'd look at your network path from the worker to the redis port, and see if theres anything that could be stopping the heatbeat connection occurring (even intermittently)

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@dmitrii-doronin commented on GitHub (Jan 14, 2026):

This issues is still relevant. Has anyone found a solution to this?

@dmitrii-doronin commented on GitHub (Jan 14, 2026): This issues is still relevant. Has anyone found a solution to this?

kerem commented

2026-03-02 05:20:18 +03:00

Author

Owner

@armistcxy commented on GitHub (Jan 14, 2026):

I have encountered this situation at some points in staging environment when I suddenly kill the pod that runs the worker.

IMO the simplest solution is writing a background job or registering a scheduled task to peforming these "crash" tasks again. (You must be careful with the state of crash tasks and idempotency by the way)

The comment from Xenov also makes sense, as I have said if it is safe to retry then just retry the task

@armistcxy commented on GitHub (Jan 14, 2026): I have encountered this situation at some points in staging environment when I suddenly kill the pod that runs the worker. IMO the simplest solution is writing a background job or registering a scheduled task to peforming these "crash" tasks again. (You must be careful with the state of crash tasks and idempotency by the way) The comment from Xenov also makes sense, as I have said if it is safe to retry then just retry the task

kerem referenced this issue

2026-03-07 22:06:20 +03:00

[GH-ISSUE #299] [BUG] cannot cancel task #1138

kerem referenced this issue

2026-03-15 19:24:03 +03:00

[GH-ISSUE #299] [BUG] cannot cancel task #2147

Rows
Columns

[GH-ISSUE #612] [BUG] asynq: task lease expired ，What does it mean？ #299