[GH-ISSUE #1160] Cron going down before they are supposed to

kerem commented

2026-02-25 23:43:37 +03:00

Owner

Originally created by @shaqaruden on GitHub (May 20, 2025).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/1160

I have some cron jobs that don't run often (Weekly, and monthly). I have a one hour grace period set on them all. It seems that for these they will down before the next cron configured schedule as lapsed.

For example I have this one cron that runs monthly, configured as so

Looking at the events I see it ping then go down 5 hours later

I verified that my cron job was setup correctly

Is this expected behavior or is this a bug?

Originally created by @shaqaruden on GitHub (May 20, 2025). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/1160 I have some cron jobs that don't run often (Weekly, and monthly). I have a one hour grace period set on them all. It seems that for these they will down before the next cron configured schedule as lapsed. For example I have this one cron that runs monthly, configured as so ![Image](https://github.com/user-attachments/assets/065c3d83-b5f7-44eb-a76f-85f5ef4e3e30) Looking at the events I see it ping then go down 5 hours later ![Image](https://github.com/user-attachments/assets/a22f453d-2884-4ccc-a48f-701f1b8061f0) I verified that my cron job was setup correctly ![Image](https://github.com/user-attachments/assets/15d52e78-8c71-403b-9a11-8bb1a846e660) Is this expected behavior or is this a bug?

kerem closed this issue

2026-02-25 23:43:37 +03:00

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@cuu508 commented on GitHub (May 20, 2025):

According to the configured schedule, Healthchecks expects a ping at or after 10:00. There is no ping between 10:00 and 11:00, so at 11:00 grace time runs out and the check's status changes to "down".

The ping at 6:04 does not count, because it arrived too early, before 10:00.

@cuu508 commented on GitHub (May 20, 2025): According to the configured schedule, Healthchecks expects a ping at or after 10:00. There is no ping between 10:00 and 11:00, so at 11:00 grace time runs out and the check's status changes to "down". The ping at 6:04 does not count, because it arrived too early, before 10:00.

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@shaqaruden commented on GitHub (May 20, 2025):

I overlooked that detail, thank you for pointing that out. The issue is the timezone in my Healthchecks container is not correct then, correct?

@shaqaruden commented on GitHub (May 20, 2025): I overlooked that detail, thank you for pointing that out. The issue is the timezone in my Healthchecks container is not correct then, correct?

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@cuu508 commented on GitHub (May 22, 2025):

The timezone of Healthchecks container should not matter (unless you are doing wonky things like mounting over /etc/localtime). I would check the following:

what is the timezone on the system sending pings? Is it indeed America/Toronto?
what is the cron schedule on the system sending pings? Is it indeed "0 10 15 * *"

@cuu508 commented on GitHub (May 22, 2025): The timezone of Healthchecks container should not matter (unless you are doing wonky things like [mounting over /etc/localtime](https://github.com/healthchecks/healthchecks/issues/1118#issuecomment-2619455516)). I would check the following: * what is the timezone on the system sending pings? Is it indeed America/Toronto? * what is the cron schedule on the system sending pings? Is it indeed "0 10 15 * *"

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@shaqaruden commented on GitHub (May 23, 2025):

The timezone on the system sending the ping is correct and it's time matches the time on the server. Yes the cron schedule is 0 10 15 * *

@shaqaruden commented on GitHub (May 23, 2025): The timezone on the system sending the ping is correct and it's time matches the time on the server. Yes the cron schedule is 0 10 15 * *

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@cuu508 commented on GitHub (May 24, 2025):

What is the timezone of the Healthchecks container?
What Healthchecks container image are you using?
Have you modified the container in any way?
What is your Healthchecks configuration (compose file, or .env file, or local_settings file)?
What is the timezone of the client system?
What is the local time (output of the date command) in the Healthchecks container right now?
What is the local time (output of the date command) on the client system right now?
What are the contents of the crontab file on the client system?
If click the "Ping" button in Healthchecks' Ping Details page, a new entry will appear in the log of events. Click on it and post the screenshot of the dialog that appears.
What is the output of journalctl --since "2025-05-15" -t CRON on the client system?

Please answer every question fully (instead of answering just the ones you think are relevant). Please edit out any sensitive information.

@cuu508 commented on GitHub (May 24, 2025): * What is the timezone of the Healthchecks container? * What Healthchecks container image are you using? * Have you modified the container in any way? * What is your Healthchecks configuration (compose file, or .env file, or local_settings file)? * What is the timezone of the client system? * What is the local time (output of the `date` command) in the Healthchecks container right now? * What is the local time (output of the `date` command) on the client system right now? * What are the contents of the crontab file on the client system? * If click the "Ping" button in Healthchecks' Ping Details page, a new entry will appear in the log of events. Click on it and post the screenshot of the dialog that appears. * What is the output of `journalctl --since "2025-05-15" -t CRON` on the client system? Please answer every question fully (instead of answering just the ones you think are relevant). Please edit out any sensitive information.

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@shaqaruden commented on GitHub (May 27, 2025):

Timezone of Healthchecks container (After updating timezone of container)
EDT
Healthchecks Image
healthchecks/healthchecks:latest
Dockerfile (Set Timezone(New after creating issue))

FROM healthchecks/healthchecks:latest

ARG TZ

USER root

ENV TZ=${TZ:-UTC}

RUN echo $TZ > /etc/timezone


USER hc

.env file

TZ=America/Toronto
ALLOWED_HOSTS=*
APPRISE_ENABLED=False
DB=postgres
DB_CONN_MAX_AGE=0
DB_HOST=db
DB_NAME=lti_admin
DB_PASSWORD=*redacted*
DB_PORT=5432
DB_SSLMODE=prefer
DB_TARGET_SESSION_ATTRS=read-write
DB_USER=postgres
DEBUG=False
DEFAULT_FROM_EMAIL=*redacted*
DISCORD_CLIENT_ID=
DISCORD_CLIENT_SECRET=
EMAIL_HOST=smtp.office365.com
EMAIL_HOST_PASSWORD=*redacted*
EMAIL_HOST_USER=*redacted*
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_USE_VERIFICATION=True
INTEGRATIONS_ALLOW_PRIVATE_IPS=False
LINENOTIFY_CLIENT_ID=
LINENOTIFY_CLIENT_SECRET=
MASTER_BADGE_LABEL="*redacted*"
MATRIX_ACCESS_TOKEN=
MATRIX_HOMESERVER=
MATRIX_USER_ID=
MATTERMOST_ENABLED=True
MSTEAMS_ENABLED=True
OPSGENIE_ENABLED=True
PAGERTREE_ENABLED=True
PD_APP_ID=
PD_ENABLED=True
PING_BODY_LIMIT=None
PING_EMAIL_DOMAIN=healthchecks.example.com
PING_ENDPOINT=http://healthchecks.example.com/ping/
PROMETHEUS_ENABLED=True
PUSHBULLET_CLIENT_ID=
PUSHBULLET_CLIENT_SECRET=
PUSHOVER_API_TOKEN=
PUSHOVER_EMERGENCY_EXPIRATION=86400
PUSHOVER_EMERGENCY_RETRY_DELAY=300
PUSHOVER_SUBSCRIPTION_URL=
REGISTRATION_OPEN=False
REMOTE_USER_HEADER=
ROCKETCHAT_ENABLED=True
RP_ID=
S3_ACCESS_KEY=
S3_BUCKET=
S3_ENDPOINT=
S3_REGION=
S3_SECRET_KEY=
S3_TIMEOUT=60
S3_SECURE=True
SECRET_KEY=*redacted*
SHELL_ENABLED=False
SIGNAL_CLI_SOCKET=
SITE_LOGO_URL="data:image/png;base64,*TOO LONG*"
SITE_NAME="*redacted*"
SITE_ROOT=http://healthchecks.example.com
SLACK_CLIENT_ID=
SLACK_CLIENT_SECRET=
SLACK_ENABLED=True
# SMTPD_PORT=
SPIKE_ENABLED=True
TELEGRAM_BOT_NAME=ExampleBot
TELEGRAM_TOKEN=
TRELLO_APP_KEY=
TWILIO_ACCOUNT=
TWILIO_AUTH=
TWILIO_FROM=
TWILIO_USE_WHATSAPP=False
USE_PAYMENTS=False
VICTOROPS_ENABLED=True
WEBHOOKS_ENABLED=True
WHATSAPP_DOWN_CONTENT_SID=
WHATSAPP_UP_CONTENT_SID=
ZULIP_ENABLED=True

Timezone of client
EDT
Local time of Healthchecks
Local time of client
content of crontab

0 10 15 * *  cd /app && php /app/SCRIPTS/5s_scheduler.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/8b574a48-1b30-4c4e-a5e8-5317dbe34c8e 1>/proc/1/fd/1 2>/proc/1/fd/2
0 3  * * 1 cd /app && php /app/SCRIPTS/5s_audit_reminder.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/43c3e792-1836-4a3f-b866-bf9412d0969c 1>/proc/1/fd/1 2>/proc/1/fd/2
0 0  * * 4 cd /app && php /app/SCRIPTS/5s_audit_monitor.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/3000193f-bfd8-4ea7-bca0-b192a419d8c1 1>/proc/1/fd/1 2>/proc/1/fd/2
0 0  * * 0 cd /app && php /app/SCRIPTS/5s_audit_cleanup.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/36bd19d0-ece2-435a-98be-af748f562f5b 1>/proc/1/fd/1 2>/proc/1/fd/2
0 0  * * 0 cd /app && php /app/SCRIPTS/5s_image_maintenance.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/2ce461b1-6f3a-4a50-8309-2ed3e5784582 1>/proc/1/fd/1 2>/proc/1/fd/2

Ping test
Journalctl output
No output to provide as client is a docker container and does not have journalctl installed

Here is an additional screenshot of the healthcheck I used for the test. As you can see it's previous ping from the client had the same effect of going down. This one does not quite make as much sense it is suppose to go off a 3AM every Monday but Healthchecks shows the same 11PM ping

@shaqaruden commented on GitHub (May 27, 2025): 1. Timezone of Healthchecks container (After updating timezone of container) EDT 2. Healthchecks Image healthchecks/healthchecks:latest 3. Dockerfile (Set Timezone(New after creating issue)) ``` FROM healthchecks/healthchecks:latest ARG TZ USER root ENV TZ=${TZ:-UTC} RUN echo $TZ > /etc/timezone USER hc ``` 4. .env file ```env TZ=America/Toronto ALLOWED_HOSTS=* APPRISE_ENABLED=False DB=postgres DB_CONN_MAX_AGE=0 DB_HOST=db DB_NAME=lti_admin DB_PASSWORD=*redacted* DB_PORT=5432 DB_SSLMODE=prefer DB_TARGET_SESSION_ATTRS=read-write DB_USER=postgres DEBUG=False DEFAULT_FROM_EMAIL=*redacted* DISCORD_CLIENT_ID= DISCORD_CLIENT_SECRET= EMAIL_HOST=smtp.office365.com EMAIL_HOST_PASSWORD=*redacted* EMAIL_HOST_USER=*redacted* EMAIL_PORT=587 EMAIL_USE_TLS=True EMAIL_USE_VERIFICATION=True INTEGRATIONS_ALLOW_PRIVATE_IPS=False LINENOTIFY_CLIENT_ID= LINENOTIFY_CLIENT_SECRET= MASTER_BADGE_LABEL="*redacted*" MATRIX_ACCESS_TOKEN= MATRIX_HOMESERVER= MATRIX_USER_ID= MATTERMOST_ENABLED=True MSTEAMS_ENABLED=True OPSGENIE_ENABLED=True PAGERTREE_ENABLED=True PD_APP_ID= PD_ENABLED=True PING_BODY_LIMIT=None PING_EMAIL_DOMAIN=healthchecks.example.com PING_ENDPOINT=http://healthchecks.example.com/ping/ PROMETHEUS_ENABLED=True PUSHBULLET_CLIENT_ID= PUSHBULLET_CLIENT_SECRET= PUSHOVER_API_TOKEN= PUSHOVER_EMERGENCY_EXPIRATION=86400 PUSHOVER_EMERGENCY_RETRY_DELAY=300 PUSHOVER_SUBSCRIPTION_URL= REGISTRATION_OPEN=False REMOTE_USER_HEADER= ROCKETCHAT_ENABLED=True RP_ID= S3_ACCESS_KEY= S3_BUCKET= S3_ENDPOINT= S3_REGION= S3_SECRET_KEY= S3_TIMEOUT=60 S3_SECURE=True SECRET_KEY=*redacted* SHELL_ENABLED=False SIGNAL_CLI_SOCKET= SITE_LOGO_URL="data:image/png;base64,*TOO LONG*" SITE_NAME="*redacted*" SITE_ROOT=http://healthchecks.example.com SLACK_CLIENT_ID= SLACK_CLIENT_SECRET= SLACK_ENABLED=True # SMTPD_PORT= SPIKE_ENABLED=True TELEGRAM_BOT_NAME=ExampleBot TELEGRAM_TOKEN= TRELLO_APP_KEY= TWILIO_ACCOUNT= TWILIO_AUTH= TWILIO_FROM= TWILIO_USE_WHATSAPP=False USE_PAYMENTS=False VICTOROPS_ENABLED=True WEBHOOKS_ENABLED=True WHATSAPP_DOWN_CONTENT_SID= WHATSAPP_UP_CONTENT_SID= ZULIP_ENABLED=True ``` 5. Timezone of client EDT 6. Local time of Healthchecks ![Image](https://github.com/user-attachments/assets/fbf189c3-12b3-48a1-bcde-de1a1dd7164c) 7. Local time of client ![Image](https://github.com/user-attachments/assets/d4502112-668f-404f-9f17-47e2b7f4a515) 8. content of crontab ``` 0 10 15 * * cd /app && php /app/SCRIPTS/5s_scheduler.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/8b574a48-1b30-4c4e-a5e8-5317dbe34c8e 1>/proc/1/fd/1 2>/proc/1/fd/2 0 3 * * 1 cd /app && php /app/SCRIPTS/5s_audit_reminder.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/43c3e792-1836-4a3f-b866-bf9412d0969c 1>/proc/1/fd/1 2>/proc/1/fd/2 0 0 * * 4 cd /app && php /app/SCRIPTS/5s_audit_monitor.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/3000193f-bfd8-4ea7-bca0-b192a419d8c1 1>/proc/1/fd/1 2>/proc/1/fd/2 0 0 * * 0 cd /app && php /app/SCRIPTS/5s_audit_cleanup.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/36bd19d0-ece2-435a-98be-af748f562f5b 1>/proc/1/fd/1 2>/proc/1/fd/2 0 0 * * 0 cd /app && php /app/SCRIPTS/5s_image_maintenance.php && curl -fsS -m 10 --retry 5 -o /dev/null http://healthchecks.example.com/ping/2ce461b1-6f3a-4a50-8309-2ed3e5784582 1>/proc/1/fd/1 2>/proc/1/fd/2 ``` 9. Ping test ![Image](https://github.com/user-attachments/assets/9e18ab09-a8fc-41ad-8df0-1c8eac101db2) 10. Journalctl output No output to provide as client is a docker container and does not have journalctl installed Here is an additional screenshot of the healthcheck I used for the test. As you can see it's previous ping from the client had the same effect of going down. This one does not quite make as much sense it is suppose to go off a 3AM every Monday but Healthchecks shows the same 11PM ping ![Image](https://github.com/user-attachments/assets/1781ab54-5c45-4e6f-a91e-9ca5c42d0345)

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@cuu508 commented on GitHub (May 29, 2025):

Thanks for the additional information and sorry for the delay getting back to you.

In the 9th step (ping test), Healthchecks shows the local time as 14:46. Was that the expected value, i.e. was the wall clock time indeed 14:46 at the time of ping? Or was the displayed time also 4 hours off?

If the timestamps look correct when pinging manually, but are 4 hours off for events triggered by cron, then the next step would be to verify if cron runs jobs at the right time.

One idea on how to test this, if feasible, would be to add another cron job:

* *  * * * curl --data-raw "$(curl -v google.com 2>&1 | grep Date)" http://healthchecks.example.com/ping/43c3e792-1836-4a3f-b866-bf9412d0969c

It runs every minute and grabs the Date response header from google.com, and submits it in POST body to the Healthchecks instance. After it has run a couple times, take a look at the ping details dialog and see if the UTC time matches up with the timestamp in the ping's body.

@cuu508 commented on GitHub (May 29, 2025): Thanks for the additional information and sorry for the delay getting back to you. In the 9th step (ping test), Healthchecks shows the local time as 14:46. Was that the expected value, i.e. was the wall clock time indeed 14:46 at the time of ping? Or was the displayed time also 4 hours off? If the timestamps look correct when pinging manually, but are 4 hours off for events triggered by cron, then the next step would be to verify if cron runs jobs at the right time. One idea on how to test this, if feasible, would be to add another cron job: ``` * * * * * curl --data-raw "$(curl -v google.com 2>&1 | grep Date)" http://healthchecks.example.com/ping/43c3e792-1836-4a3f-b866-bf9412d0969c ``` It runs every minute and grabs the Date response header from google.com, and submits it in POST body to the Healthchecks instance. After it has run a couple times, take a look at the ping details dialog and see if the UTC time matches up with the timestamp in the ping's body.

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@shaqaruden commented on GitHub (May 29, 2025):

I can definitely do that. I will add that cron tomorrow morning

@shaqaruden commented on GitHub (May 29, 2025): I can definitely do that. I will add that cron tomorrow morning

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@shaqaruden commented on GitHub (May 30, 2025):

Here is a screenshot of a ping from that cron

@shaqaruden commented on GitHub (May 30, 2025): Here is a screenshot of a ping from that cron ![Image](https://github.com/user-attachments/assets/11dbe5f1-490d-4c38-98da-38f371a66f22)

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@cuu508 commented on GitHub (May 30, 2025):

Thanks. This looks as expected, the time in google's "Date:" header matches the UTC time of the ping.

Can you do another experiment? Add a job that should run 5 minutes in the future. If the current time is 12:34, add a job

39 12  * * * curl --data-raw "this should run at 12:39" http://healthchecks.example.com/ping/43c3e792-1836-4a3f-b866-bf9412d0969c

Then, after 12:39, see if a ping was sent, and whether its timestamp is 12:39 EDT (16:39 UTC).

The idea with this is to check if the cron daemon runs the job at the correct time.

@cuu508 commented on GitHub (May 30, 2025): Thanks. This looks as expected, the time in google's "Date:" header matches the UTC time of the ping. Can you do another experiment? Add a job that should run 5 minutes in the future. If the current time is 12:34, add a job ``` 39 12 * * * curl --data-raw "this should run at 12:39" http://healthchecks.example.com/ping/43c3e792-1836-4a3f-b866-bf9412d0969c ``` Then, after 12:39, see if a ping was sent, and whether its timestamp is 12:39 EDT (16:39 UTC). The idea with this is to check if the cron daemon runs the job at the correct time.

kerem commented

2026-02-25 23:43:38 +03:00

Author

Owner

@cuu508 commented on GitHub (Jun 13, 2025):

No response, closing.

@cuu508 commented on GitHub (Jun 13, 2025): No response, closing.

Rows
Columns

[GH-ISSUE #1160] Cron going down before they are supposed to #800