starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-26 07:25:56 +03:00

[GH-ISSUE #710] Question: A way to gracefully shutdown server after single task execution #2370

New issue

Open

opened 2026-03-15 20:16:33 +03:00 by kerem · 5 comments

kerem commented

2026-03-15 20:16:33 +03:00

Owner

Originally created by @romanilchyshyn on GitHub (Jul 30, 2023).
Original GitHub issue: https://github.com/hibiken/asynq/issues/710

I want to run asynq.Server-s on top of pool of machines (vm-s, containers whatever). Each machine runs single server with Concurrency = 1. Once a task is successfully executed, I want to call Shutdown() on the server to stop it from processing further tasks. (In general I want to terminate machine and start new "clean" machine from pool manager).

However, I have encountered an issue where I cannot simply signal the server to shut down from the task handler. If I proceed with such a shutdown, the handleSucceededMessage(..) will not be called, and the task will remain in the queue.

So, is there any way to implement such behavior?

Originally created by @romanilchyshyn on GitHub (Jul 30, 2023). Original GitHub issue: https://github.com/hibiken/asynq/issues/710 I want to run `asynq.Server`-s on top of pool of machines (vm-s, containers whatever). Each machine runs single server with `Concurrency = 1`. Once a task is successfully executed, I want to call Shutdown() on the server to stop it from processing further tasks. (In general I want to terminate machine and start new "clean" machine from pool manager). However, I have encountered an issue where I cannot simply signal the server to shut down from the task handler. If I proceed with such a shutdown, the handleSucceededMessage(..) will not be called, and the task will remain in the queue. So, is there any way to implement such behavior?

kerem added the

question

label

2026-03-15 20:16:33 +03:00

kerem commented

2026-03-15 20:16:47 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Jul 31, 2023):

Are you setting Config.ShutdownTimeout to be atleast ~ time taken to execute the task? You will also need to configure your vm/container orchestrator e.g. Docker Compose not to send a SIGKILL instead set a timeout also on that. That is the safest graceful shutdown approach I can think of.

@kamikazechaser commented on GitHub (Jul 31, 2023): Are you setting `Config.ShutdownTimeout` to be atleast ~ time taken to execute the task? You will also need to configure your vm/container orchestrator e.g. Docker Compose not to send a SIGKILL instead set a timeout also on that. That is the safest graceful shutdown approach I can think of.

kerem commented

2026-03-15 20:16:52 +03:00

Author

Owner

@romanilchyshyn commented on GitHub (Jul 31, 2023):

Thanks for the response. But I afraid timing out is not what I'm looking for.
I prob explained it badly. My aim is:

start program, start server
server picks single task from redis and executes it
somehow understand that task has been executed and call server Shutdown before start processing next task from redis
exit program

Don't care about how it orchestrated from the outside.

@romanilchyshyn commented on GitHub (Jul 31, 2023): Thanks for the response. But I afraid timing out is not what I'm looking for. I prob explained it badly. My aim is: - start program, start server - server picks single task from redis and executes it - somehow understand that task has been executed and call server `Shutdown` before start processing next task from redis - exit program Don't care about how it orchestrated from the outside.

kerem commented

2026-03-15 20:16:57 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Aug 1, 2023):

I see. Even if you were to set a concurrency of 1 on the asynq server, I don't see any guarantee that the next task won't be dequeued before it is shutdown unless you do some complex signal processing.

Though if you want such a setup why not use a simple RPUSH to queue and LPOP by a custom executor. That way you will have a lot of control.

@kamikazechaser commented on GitHub (Aug 1, 2023): I see. Even if you were to set a concurrency of 1 on the asynq server, I don't see any guarantee that the next task won't be dequeued before it is shutdown unless you do some complex signal processing. Though if you want such a setup why not use a simple RPUSH to queue and LPOP by a custom executor. That way you will have a lot of control.

kerem commented

2026-03-15 20:17:02 +03:00

Author

Owner

@voidshard commented on GitHub (Oct 14, 2023):

I have another question on graceful shutdown that I think is related to this; The server has a Stop() and a Shutdown(), from reading the comments Stop() prevents the server from picking up more tasks and Shutdown() well .. shutsdown. In theory one can call Stop() on a server, then wait until that server has finished processing all it's tasks, then call Shutdown().

Looking at the inspector I can see a Servers() call which returns all servers, and serverinfos include an ID. But I can't see how I can get the ID of the server I called Stop() on ..?

So in practice, I'm unclear how I would check that the server I've just called Stop() on is actually ready for Shutdown(). I mean, sure I could set some kind of max task time and wait that long before shutdown .. but that seems a little weird.

I'd like to request:

calling NewServer should yield a way to get a server ID
ideally, we'd have a way to GetServerInfo(serverID string)
.. Happy to add myself if you want.

My apologies if there is a way to do this currently and I haven't found it!

@voidshard commented on GitHub (Oct 14, 2023): I have another question on graceful shutdown that I think is related to this; The server has a Stop() and a Shutdown(), from reading the comments Stop() prevents the server from picking up more tasks and Shutdown() well .. shutsdown. In theory one can call Stop() on a server, then wait until that server has finished processing all it's tasks, then call Shutdown(). Looking at the inspector I can see a Servers() call which returns all servers, and serverinfos include an ID. But I can't see how I can get the ID of the server I called Stop() on ..? So in practice, I'm unclear how I would check that the server I've just called Stop() on is actually ready for Shutdown(). I mean, sure I could set some kind of max task time and wait that long before shutdown .. but that seems a little weird. I'd like to request: - calling NewServer should yield a way to get a server ID - ideally, we'd have a way to GetServerInfo(serverID string) .. Happy to add myself if you want. My apologies if there is a way to do this currently and I haven't found it!

kerem commented

2026-03-15 20:17:07 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Oct 19, 2023):

@voidshard Moved your question to a new issue.

@kamikazechaser commented on GitHub (Oct 19, 2023): @voidshard Moved your question to a new issue.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2370

No description provided.

Rows
Columns