[GH-ISSUE #815] My checks started to fail after the grace period, even when they ran.

kerem commented

2026-02-25 23:42:54 +03:00

Owner

Originally created by @yurividal on GitHub (Mar 29, 2023).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/815

My healtchecks app has been running fine for months. SInce last week, it started behaving differently.
Some checks are being marked as down after the grace period, even after they successfuly ran.

Example:

The sync above runs every weekday hours. There is a 2 hour grace period. The script successfuly runs at 6pm. But, the check goes to down at exactly 8pm. (eventhough you can clearly see in the image that the last ping was 3 hour ago which is less than 24h.).

I believe this started on the day our clocks changed to DST, but i'm not sure if this is related.
I am seeing this same behaviour with a few of my checks, but not all.

What could be wrong?!

Originally created by @yurividal on GitHub (Mar 29, 2023). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/815 My healtchecks app has been running fine for months. SInce last week, it started behaving differently. Some checks are being marked as down after the grace period, even after they successfuly ran. Example: <img width="877" alt="image" src="https://user-images.githubusercontent.com/8555749/228655921-3a76c615-611a-4ed9-8cb3-195f9d6c5569.png"> The sync above runs every weekday hours. There is a 2 hour grace period. The script successfuly runs at 6pm. But, the check goes to down at exactly 8pm. (eventhough you can clearly see in the image that the last ping was 3 hour ago which is less than 24h.). I believe this started on the day our clocks changed to DST, but i'm not sure if this is related. I am seeing this same behaviour with a few of my checks, but not all. What could be wrong?!

kerem closed this issue

2026-02-25 23:42:54 +03:00

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@yurividal commented on GitHub (Mar 29, 2023):

This only seems to affect cron-based checks

@yurividal commented on GitHub (Mar 29, 2023): This only seems to affect cron-based checks

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@yurividal commented on GitHub (Mar 29, 2023):

Ok, this seems to be related to this

Indeed my host had BST time (British summer time, London) and the container had UTC time. I have applied ENV TZ=Europe/London and now the container shows BST.

Let's see if this will solve it.

@yurividal commented on GitHub (Mar 29, 2023): Ok, this seems to be related to [this](https://github.com/healthchecks/healthchecks/issues/704) Indeed my host had BST time (British summer time, London) and the container had UTC time. I have applied ENV TZ=Europe/London and now the container shows BST. Let's see if this will solve it.

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@thestraycat commented on GitHub (Apr 6, 2023):

I'm the original poster of that thread... I've never managed to fix it... It's so annoying. :(

My linuxserver.io container (healthchecks) timezone = Thu Apr 6 22:21:28 BST 2023
My container host (unraid) timezone = Thu Apr 6 22:21:51 BST 2023
My client sending the cron job (a raspberry pi) = Thu 6 Apr 22:21:00 BST 2023

There is no BST option to set within healthchecks, so i've manually added a variable to force Europe/London. The cron job settings in heathchecks are also set to Europe/London

When i SSH to my rpi client that runs the cron jobs and i curl a test to the healthchecks.io container and check the timestamp on both the start and end pings that are recieved i see them both display as 9.20pm recieved. (Although the time in the UK is 10:23pm)

But then if i click on the 'show events' button on the same context window, which is ping #452 it now shows up as: 10:24pm it's like theres an internal conflict in healthchecks.

It renders the whole thing unusable. Such a shame, i really like the project. :(

I don't know what else to do, all my devices, host, healthcehcks container and client are all set to BST. The cron job is setup as Europe/London. But yet there is always an hours worth of time difference. :(

I've just tried setting the grace time of my 5 second jobs too 70 minutes to see if they finish and dont hang.

@thestraycat commented on GitHub (Apr 6, 2023): I'm the original poster of that thread... I've never managed to fix it... It's so annoying. :( My linuxserver.io container (healthchecks) timezone = Thu Apr 6 22:21:28 BST 2023 My container host (unraid) timezone = Thu Apr 6 22:21:51 BST 2023 My client sending the cron job (a raspberry pi) = Thu 6 Apr 22:21:00 BST 2023 There is no BST option to set within healthchecks, so i've manually added a variable to force Europe/London. The cron job settings in heathchecks are also set to **Europe/London** When i SSH to my rpi client that runs the cron jobs and i curl a test to the healthchecks.io container and check the timestamp on both the start and end pings that are recieved i see them both display as 9.20pm recieved. (Although the time in the UK is 10:23pm) ![image](https://user-images.githubusercontent.com/9296295/230506700-08519e52-c042-463b-aebc-e9234fb1459a.png) But then if i click on the 'show events' button on the same context window, which is ping #452 it now shows up as: 10:24pm it's like theres an internal conflict in healthchecks. ![image](https://user-images.githubusercontent.com/9296295/230506765-a83f0481-0717-45aa-9083-a6e96e4eb5f1.png) It renders the whole thing unusable. Such a shame, i really like the project. :( I don't know what else to do, all my devices, host, healthcehcks container and client are all set to BST. The cron job is setup as Europe/London. But yet there is always an hours worth of time difference. :( I've just tried setting the grace time of my 5 second jobs too 70 minutes to see if they finish and dont hang.

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@cuu508 commented on GitHub (Apr 11, 2023):

I don't know what else to do, all my devices, host, healthcehcks container and client are all set to BST. The cron job is setup as Europe/London. But yet there is always an hours worth of time difference. :(

My recommendation:

set the host system to use UTC
use the official container image
do not touch any timezone settings inside the container (they should be UTC, same as the base system)

@cuu508 commented on GitHub (Apr 11, 2023): > I don't know what else to do, all my devices, host, healthcehcks container and client are all set to BST. The cron job is setup as Europe/London. But yet there is always an hours worth of time difference. :( My recommendation: * set the host system to use UTC * use [the official container image](https://hub.docker.com/r/healthchecks/healthchecks) * do not touch any timezone settings inside the container (they should be UTC, same as the base system)

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@yurividal commented on GitHub (Apr 11, 2023):

If your healthchecks is behind a reverse proxy, I think you also need to check the timezone on your proxy host o container.

@yurividal commented on GitHub (Apr 11, 2023): If your healthchecks is behind a reverse proxy, I think you also need to check the timezone on your proxy host o container.

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@thestraycat commented on GitHub (Apr 11, 2023):

@cuu508 - The Host machine is running UTC. Im assuming BST (is simply british summer time, so UTC+1? when the dalight saving button is ticked?) the host shows correct time at time of writing.

I could move over the official container and maybe i'll try it in the next few days, but i'm very reulctant as its a full rebuild as the container variables are likely different, and then it means regenerating all of my UUID's for the current checks as their auto-generated by healthchecks and cant be reditted easily in the healthchekcs etc. Then it means editting all the scripts and cronjob to reflect it. So i'd rather not jump to that really.

@yurividal - Good shout. Just had a look at the SWAG config and it shows the same as all the other parts of the chain (BST)

root@1e0b5bf97432:/# date
Tue Apr 11 11:54:47 BST 2023

One thing i will note is that, i started to use internal ip addresses for my internal checks, so healthchecks will normally auto generate me an external URL link using my domain For example, which is what is set up by healthchecks in healthchecks

https://healthchecks.mydomain.co.uk/ping/5aaaf570-fcf5-4fa9-8444-c03d601df13a

But i generally like to keep the check internal, and not have my internet connection as a possible point of failure for these particular cron checks so i use my internal IP addressing which seems to be recieved just fine. So have always used it like this.

I take it thats ok?

http://192.168.1.125:8000/ping/5aaaf570-fcf5-4fa9-8444-c03d601df13a

@thestraycat commented on GitHub (Apr 11, 2023): @cuu508 - The Host machine is running UTC. Im assuming BST (is simply british summer time, so UTC+1? when the dalight saving button is ticked?) the host shows correct time at time of writing. ![image](https://user-images.githubusercontent.com/9296295/231191639-e0244f17-1039-49c9-935a-921fa7e7bf82.png) I could move over the official container and maybe i'll try it in the next few days, but i'm very reulctant as its a full rebuild as the container variables are likely different, and then it means regenerating all of my UUID's for the current checks as their auto-generated by healthchecks and cant be reditted easily in the healthchekcs etc. Then it means editting all the scripts and cronjob to reflect it. So i'd rather not jump to that really. @yurividal - Good shout. Just had a look at the SWAG config and it shows the same as all the other parts of the chain (BST) root@1e0b5bf97432:/# date Tue Apr 11 11:54:47 BST 2023 One thing i will note is that, i started to use internal ip addresses for my internal checks, so healthchecks will normally auto generate me an external URL link using my domain For example, which is what is set up by healthchecks in healthchecks https://healthchecks.mydomain.co.uk/ping/5aaaf570-fcf5-4fa9-8444-c03d601df13a But i generally like to keep the check internal, and not have my internet connection as a possible point of failure for these particular cron checks so i use my internal IP addressing which seems to be recieved just fine. So have always used it like this. I take it thats ok? http://192.168.1.125:8000/ping/5aaaf570-fcf5-4fa9-8444-c03d601df13a

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@cuu508 commented on GitHub (Apr 11, 2023):

I could move over the official container and maybe i'll try it in the next few days, but i'm very reulctant as its a full rebuild as the container variables are likely different, and then it means regenerating all of my UUID's for the current checks

You should be able to keep using the existing database.

If you're using Postgres or MySQL, it's as easy as specifying the same DB_ env variables for the new container as the old one.
If you're using sqlite on a volume, attach the same volume to the new container
If you're using sqlite, and the database file is stored inside the container, you can copy the database file from the old container to a safe location and then to the new container

Either way, make sure to backup the database before you start moving stuff!

@cuu508 commented on GitHub (Apr 11, 2023): > I could move over the official container and maybe i'll try it in the next few days, but i'm very reulctant as its a full rebuild as the container variables are likely different, and then it means regenerating all of my UUID's for the current checks You should be able to keep using the existing database. * If you're using Postgres or MySQL, it's as easy as specifying the same DB_ env variables for the new container as the old one. * If you're using sqlite on a volume, attach the same volume to the new container * If you're using sqlite, and the database file is stored inside the container, you can copy the database file from the old container to a safe location and then to the new container Either way, make sure to backup the database before you start moving stuff!

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@thestraycat commented on GitHub (Apr 11, 2023):

@cuu508 - Thanks, i believe the DB is baked into the image. I dont see any external mappings to a seperate database container. I'm sure it's doable, but im not a DB guy and can see that going wrong, i mean linuxserver.io are probably the biggest container maintainer in the scene, so i'd think if the image was funky there would be more people maybe mentioning it? But yes. - I'll try with the official image when time permits in the next few days. Good shout.

I've since done some more checks and have tried initiating the healthchecks cron call from the container host, another container on the host and my windows pc using the linux subsystem (WSL). And healthchecks recieves all 3 of the cron checks all at 2:39pm (it's 3:39pm currently) it seems no matter where i send from healthchecks reports recieving the curl request an hour before current time.

@thestraycat commented on GitHub (Apr 11, 2023): @cuu508 - Thanks, i believe the DB is baked into the image. I dont see any external mappings to a seperate database container. I'm sure it's doable, but im not a DB guy and can see that going wrong, i mean linuxserver.io are probably the biggest container maintainer in the scene, so i'd think if the image was funky there would be more people maybe mentioning it? But yes. - I'll try with the official image when time permits in the next few days. Good shout. I've since done some more checks and have tried initiating the healthchecks cron call from the container host, another container on the host and my windows pc using the linux subsystem (WSL). And healthchecks recieves all 3 of the cron checks all at 2:39pm (it's 3:39pm currently) it seems no matter where i send from healthchecks reports recieving the curl request an hour before current time. ![image](https://user-images.githubusercontent.com/9296295/231198625-0eaf847b-0d38-4b6b-9f2f-80014d8eef26.png)

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@cuu508 commented on GitHub (Apr 11, 2023):

Note that in the above screenshot, it shows the timestamp in UTC (see the +00:00 timezone offset in the timestamp). 14:38 in UTC would be 15:38 BST.

@cuu508 commented on GitHub (Apr 11, 2023): Note that in the above screenshot, it shows the timestamp in UTC (see the +00:00 timezone offset in the timestamp). 14:38 in UTC would be 15:38 BST.

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@thestraycat commented on GitHub (Apr 11, 2023):

@cuu508 Yup. But why does it show it in UTC? And not my local time?

My timezone is set as Europe/London on the healthchecks container.
My container host is set as UTC+00:00 using NTP and shows the current time correctly here in the UK.
This particular cron check in healthchecks is setup as Europe/London.

So why show me the time recieved as 14:38?

If i change the timezone on the cron job to say UTC, it looks correct in the job at the tim as highlighted below.

However when i then go and send a curl to that particular job you can see in the history it still says it recieved it at the wrong time (Shows 16:46pm and its 17:46pm)

However if i click on the "Show Earlier events button on that same screenshot. It shows the correct time of 17:46pm for ping #483. Makes no sense. That hour difference means all my checks fails. One sceens says one thing, the next screen says something else.

@thestraycat commented on GitHub (Apr 11, 2023): @cuu508 Yup. But why does it show it in UTC? And not my local time? My timezone is set as Europe/London on the healthchecks container. My container host is set as UTC+00:00 using NTP and shows the current time correctly here in the UK. This particular cron check in healthchecks is setup as Europe/London. ![image](https://user-images.githubusercontent.com/9296295/231231506-64b9ecc5-e1f2-4c23-af67-4ae24678d4b8.png) So why show me the time recieved as 14:38? If i change the timezone on the cron job to say UTC, it looks correct in the job at the tim as highlighted below. ![image](https://user-images.githubusercontent.com/9296295/231232431-07d641fe-04e3-4c96-aca9-f6efefdc19b1.png) However when i then go and send a curl to that particular job you can see in the history it still says it recieved it at the wrong time (Shows 16:46pm and its 17:46pm) ![image](https://user-images.githubusercontent.com/9296295/231233004-ff3b4653-d07b-417f-bc09-5adef06e220b.png) However if i click on the "Show Earlier events button on that same screenshot. It shows the correct time of 17:46pm for ping #483. Makes no sense. That hour difference means all my checks fails. One sceens says one thing, the next screen says something else. ![image](https://user-images.githubusercontent.com/9296295/231233539-e91b019b-b961-44f0-a78d-ebb4e420d34b.png)

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@thestraycat commented on GitHub (Apr 11, 2023):

Could there be a discrepency between the time that the database is set too and the time that healthchecks is referencing? Or does healthchecks have an error where the daylight savings dosn't track correctly throught the UI?

I've just set the timezone of the container to Etc/UTC and left the cron job set to UTC in healthchecks and it all displays the same as above. Exactly the same. . Could it be that wherever this page is getting it's time recieved from it only ever shows the recieved time as UTC? I can't set anything anywhere it seems to make the screenshot below not show as +00:00

@thestraycat commented on GitHub (Apr 11, 2023): Could there be a discrepency between the time that the database is set too and the time that healthchecks is referencing? Or does healthchecks have an error where the daylight savings dosn't track correctly throught the UI? I've just set the timezone of the container to Etc/UTC and left the cron job set to UTC in healthchecks and it all displays the same as above. Exactly the same. . Could it be that wherever this page is getting it's time recieved from it only ever shows the recieved time as UTC? I can't set anything anywhere it seems to make the screenshot below not show as +00:00 ![image](https://user-images.githubusercontent.com/9296295/231236474-a24d4a58-538d-4f04-bdcb-25fd38371bfb.png)

kerem commented

2026-02-25 23:42:54 +03:00

Author

Owner

@cuu508 commented on GitHub (Apr 11, 2023):

I don't see discrepancies in the above screenshots.

In the database, we store timezone-aware datetimes, as is standard practice. At the display time we can convert a datetime to UTC, BST, or any other needed timezone.

In the "ping details" dialog we show "Time Received" as ISO8601 timestamp with UTC timezone. Container's timezone or host system's timezone does not influence this.

In this dialog, on the left we show a formatted date using whatever timezone the user has selected at thte top. On the right we show ISO8601 timestamp with UTC timezone.

Here by default we show dates and times in browser's timezone, but at the top right there is a switcher for switching to UTC.

@cuu508 commented on GitHub (Apr 11, 2023): I don't see discrepancies in the above screenshots. In the database, we store timezone-aware datetimes, as is standard practice. At the display time we can convert a datetime to UTC, BST, or any other needed timezone. ![image](https://user-images.githubusercontent.com/661859/231237205-3a695189-46e1-4ca3-a633-828e73713972.png) In the "ping details" dialog we show "Time Received" as ISO8601 timestamp with UTC timezone. Container's timezone or host system's timezone does not influence this. --- ![image](https://user-images.githubusercontent.com/661859/231237666-b13e16c7-412e-4fa5-87a2-2b2aec40d68b.png) In this dialog, on the left we show a formatted date using whatever timezone the user has selected at thte top. On the right we show ISO8601 timestamp with UTC timezone. --- ![image](https://user-images.githubusercontent.com/661859/231237912-fca83082-47ae-4963-b54a-c4ca1a13b732.png) Here by default we show dates and times in browser's timezone, but at the top right there is a switcher for switching to UTC.

Rows
Columns

[GH-ISSUE #815] My checks started to fail after the grace period, even when they ran. #572