starred/asynq

Fork 0

mirror of https://github.com/hibiken/asynq.git synced 2026-04-25 23:15:51 +03:00

[GH-ISSUE #820] Why are we cleaning the cron entries on shutdown? #2426

New issue

Open

opened 2026-03-15 20:27:50 +03:00 by kerem · 6 comments

kerem commented

2026-03-15 20:27:50 +03:00

Owner

Originally created by @Kenan7 on GitHub (Feb 4, 2024).
Original GitHub issue: https://github.com/hibiken/asynq/issues/820

Originally assigned to: @hibiken on GitHub.

We have a few different types of periodic jobs running (monthly, weekly, daily).

But every time our services (pods) restart, the progress for these jobs is lost, and it has to start from the beginning,

I was researching this and came across this code, and I wanted to know how relevant is it, and why it was intended like this. Maybe it was difficult to handle resuming so it was cut off until it was fixed.

github.com/hibiken/asynq@6529a1e0b1

In general, how can we resolve losing the progress of these entries?

Originally created by @Kenan7 on GitHub (Feb 4, 2024). Original GitHub issue: https://github.com/hibiken/asynq/issues/820 Originally assigned to: @hibiken on GitHub. We have a few different types of periodic jobs running (monthly, weekly, daily). But every time our services (pods) restart, the progress for these jobs is lost, and it has to start from the beginning, I was researching this and came across this code, and I wanted to know how relevant is it, and why it was intended like this. Maybe it was difficult to handle resuming so it was cut off until it was fixed. <img width="942" alt="image" src="https://github.com/hibiken/asynq/assets/11595462/06717013-20f4-4452-993e-ef0bf219745d"> https://github.com/hibiken/asynq/commit/6529a1e0b1289d01f13229f450a5a0904e162a2c In general, how can we resolve losing the progress of these entries?

kerem added the

bug

investigate

labels

2026-03-15 20:27:50 +03:00

kerem commented

2026-03-15 20:28:05 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Feb 4, 2024):

clearHistory ultimately clears the enqueue event history of all scheduler entries, I don't think this should be an issue as it just prevents the db from growing uncessarily. The heartbeater does clears the scheduler entries during shutdown. I am assuming the task entries are somehow utilized within your code?

Could you describe how you expected the scheduler to work? Are these long running jobs?

@kamikazechaser commented on GitHub (Feb 4, 2024): clearHistory ultimately clears the **enqueue event history** of all scheduler entries, I don't think this should be an issue as it just prevents the db from growing uncessarily. The heartbeater does clears the scheduler entries during shutdown. I am assuming the task entries are somehow utilized within your code? Could you describe how you expected the scheduler to work? Are these long running jobs?

kerem commented

2026-03-15 20:28:10 +03:00

Author

Owner

@Kenan7 commented on GitHub (Feb 4, 2024):

it just prevents the db from growing necessarily

I understand this, but maybe we should add some kind of condition on not removing the queued and unfinished jobs because we lose the progress?

I am assuming the task entries are somehow utilized within your code?

Also, what do you mean by this?

So we are leveraging this usage in asynq

Let's say for the daily periodic job, there needs to be a job that is handled after 55 minutes, the pod restarts, now it is beginning to calculate again for 24 hour, it is not going to be triggered after 55 minutes anymore.

The alternative is to use fixed times like everyday at 6:00 PM or something but it just does not suit the business needs.

@Kenan7 commented on GitHub (Feb 4, 2024): > it just prevents the db from growing necessarily I understand this, but maybe we should add some kind of condition on not removing the queued and unfinished jobs because we lose the progress? > I am assuming the task entries are somehow utilized within your code? Also, what do you mean by this? --- So we are leveraging this usage in asynq <img width="493" alt="image" src="https://github.com/hibiken/asynq/assets/11595462/e2495753-8368-4548-a856-a88ce50bcfc9"> Let's say for the daily periodic job, there needs to be a job that is handled after 55 minutes, the pod restarts, now it is beginning to calculate again for 24 hour, it is not going to be triggered after 55 minutes anymore. The alternative is to use fixed times like everyday at 6:00 PM or something but it just does not suit the business needs.

kerem commented

2026-03-15 20:28:15 +03:00

Author

Owner

@kamikazechaser commented on GitHub (Feb 5, 2024):

There needs to be a job that is handled after 55 minutes, the pod restarts, now it is beginning to calculate again for 24 hour, it is not going to be triggered after 55 minutes anymore.

Thanks for clarifying. I'll investigate and try and reproduce this issue.

@kamikazechaser commented on GitHub (Feb 5, 2024): > There needs to be a job that is handled after 55 minutes, the pod restarts, now it is beginning to calculate again for 24 hour, it is not going to be triggered after 55 minutes anymore. Thanks for clarifying. I'll investigate and try and reproduce this issue.

kerem commented

2026-03-15 20:28:20 +03:00

Author

Owner

@Haji-sudo commented on GitHub (Feb 27, 2024):

Did you find a way to prevent deletion? @Kenan7 @kamikazechaser

@Haji-sudo commented on GitHub (Feb 27, 2024): Did you find a way to prevent deletion? @Kenan7 @kamikazechaser

kerem commented

2026-03-15 20:28:25 +03:00

Author

Owner

@Kenan7 commented on GitHub (Feb 27, 2024):

Unfortunately. @Haji-sudo

@Kenan7 commented on GitHub (Feb 27, 2024): Unfortunately. @Haji-sudo

kerem commented

2026-03-15 20:28:31 +03:00

Author

Owner

@lsgndln commented on GitHub (Feb 27, 2024):

Same issue here, we have exactly the same use case with monthly / weekly jobs. We don't mind losing progress of ongoing jobs, as we keep track of the progress at a job level. But we would expect the job to run again on startup, as it has not been finished successfully.

This is especially visible with long-duration jobs running like so @every week or @every month. We have jobs running for hours or even days. Let's say that after 1 day the server restarts, we would have to wait for another week or month before re-running the job again.

@lsgndln commented on GitHub (Feb 27, 2024): Same issue here, we have exactly the same use case with monthly / weekly jobs. We don't mind losing progress of ongoing jobs, as we keep track of the progress at a job level. But we would expect the job to run again on startup, as it has not been finished successfully. This is especially visible with long-duration jobs running like so `@every week` or `@every month`. We have jobs running for hours or even days. Let's say that after 1 day the server restarts, we would have to wait for another week or month before re-running the job again.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/asynq#2426

No description provided.

Rows
Columns