[GH-ISSUE #510] Keep alerting when a check is down

kerem commented

2026-02-25 23:42:13 +03:00

Owner

Originally created by @kerenskybr on GitHub (May 7, 2021).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/510

Hi. How could be possible to implement a way to keep sending alerts for down checks? I tried change some things in sendlalerts.py but was not able to put this feature. Any tips?

Originally created by @kerenskybr on GitHub (May 7, 2021). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/510 Hi. How could be possible to implement a way to keep sending alerts for down checks? I tried change some things in `sendlalerts.py` but was not able to put this feature. Any tips?

kerem

2026-02-25 23:42:13 +03:00

closed this issue
added the
feature
label

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@cuu508 commented on GitHub (May 8, 2021):

Hi @kerenskybr , do you know about the hourly / daily reminders in Account Settings > Email Reports?

If you select "Remind me hourly", you will get an email message every hour as long as any check is down. A couple notes:

this feature only works with email. There currently isn't an option to receive repeated e.g. Slack alerts
this is an account-wide setting. If you have access to multiple projects, you will get repeated notifications when any check in any project is down

@cuu508 commented on GitHub (May 8, 2021): Hi @kerenskybr , do you know about the hourly / daily reminders in Account Settings > Email Reports? ![image](https://user-images.githubusercontent.com/661859/117535499-bbaa8280-affe-11eb-91ab-c10e30b36337.png) If you select "Remind me hourly", you will get an email message every hour as long as any check is down. A couple notes: - this feature only works with email. There currently isn't an option to receive repeated e.g. Slack alerts - this is an account-wide setting. If you have access to multiple projects, you will get repeated notifications when any check *in any project* is down

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@kerenskybr commented on GitHub (May 10, 2021):

Yes, for slack/webhooks alerts. I think it's a cool to have feature. I tried to implement by my self (i been using it to control backups, servers and so), with no luck.
My idea is to keep receiving the pings when a check is down (i.e. each 5 minutes)

@kerenskybr commented on GitHub (May 10, 2021): Yes, for slack/webhooks alerts. I think it's a cool to have feature. I tried to implement by my self (i been using it to control backups, servers and so), with no luck. My idea is to keep receiving the pings when a check is down (i.e. each 5 minutes)

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@cuu508 commented on GitHub (Aug 10, 2022):

Here's an UI mockup of the Account Settings – Email Reports screen:

It adds checkboxes that let the user select which configured notification channels to use for ongoing reminders. It also adds a "Every 10 minutes" option.

A couple notes:

Only few specific types of channels would show up here – email, Slack, perhaps a few others.
This would be a project-global setting. There would be no way to enable recurring notification for a single check, or for a single check-channel combination.

@rwjack, @kerenskybr what do you think, would this make sense to you?

@cuu508 commented on GitHub (Aug 10, 2022): Here's an UI mockup of the Account Settings – Email Reports screen: ![image](https://user-images.githubusercontent.com/661859/183889529-4b97eeba-bde7-4237-b40a-c248ac5fbbba.png) It adds checkboxes that let the user select which configured notification channels to use for ongoing reminders. It also adds a "Every 10 minutes" option. A couple notes: * Only few specific types of channels would show up here – email, Slack, perhaps a few others. * This would be a project-global setting. There would be no way to enable recurring notification for a single check, or for a single check-channel combination. @rwjack, @kerenskybr what do you think, would this make sense to you?

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@rwjack commented on GitHub (Aug 10, 2022):

Awesome, that looks like the thing we're looking for!

PS. I don't want to sound like a dick, but I'm just curious. Is there a reason we couldn't set the reminder timer to X minutes, instead of it being hardcoded 10/60/1440?

I hope you see that most of the people, dare I say everyone that uses healthchecks.io isn't an end user, and doesn't require simplicity. I personally wouldn't be taking the Apple route of simplicity here.

I think HC.io requires customizability and interoperability, and should get raw dog dirty with the possible configuration options. I also understand that isn't as easy to implement. Though from the perspective of my basic dev knowledge, I'd say if you've built ALL of this, passing a text box variable to a timer function presents a piece of cake for you. It's just a matter of what direction YOU want to lead the project.

@rwjack commented on GitHub (Aug 10, 2022): Awesome, that looks like the thing we're looking for! PS. I don't want to sound like a dick, but I'm just curious. Is there a reason we couldn't set the reminder timer to X minutes, instead of it being hardcoded 10/60/1440? I hope you see that most of the people, dare I say everyone that uses healthchecks.io isn't an end user, and doesn't require simplicity. I personally wouldn't be taking the Apple route of simplicity here. I think HC.io requires customizability and interoperability, and should get raw dog dirty with the possible configuration options. I also understand that isn't as easy to implement. Though from the perspective of my *basic* dev knowledge, I'd say if you've built ALL of this, passing a text box variable to a timer function presents a piece of cake for you. It's just a matter of what direction YOU want to lead the project.

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@rwjack commented on GitHub (Aug 10, 2022):

Only few specific types of channels would show up here – email, Slack, perhaps a few others.

Small remark, I use matrix, though in my original feature request, I mentioned slack as just an example. Do you have any plans on implementing matrix recurring notifications?

@rwjack commented on GitHub (Aug 10, 2022): > Only few specific types of channels would show up here – email, Slack, perhaps a few others. Small remark, I use matrix, though in my original feature request, I mentioned slack as just an example. Do you have any plans on implementing matrix recurring notifications?

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@cuu508 commented on GitHub (Aug 10, 2022):

Thanks for the feedback @rwjack.

Oops, I screwed up the mockup: the "Email Reports" screen is in the account settings, not project settings. So it would be confusing to list notification methods from individual projects here. Back to the drawing board :-)

@cuu508 commented on GitHub (Aug 10, 2022): Thanks for the feedback @rwjack. Oops, I screwed up the mockup: the "Email Reports" screen is in the account settings, not project settings. So it would be confusing to list notification methods from individual projects here. Back to the drawing board :-)

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@cuu508 commented on GitHub (Aug 10, 2022):

Is there a reason we couldn't set the reminder timer to X minutes, instead of it being hardcoded 10/60/1440?

If we can pick 2-4 choices that would satisfy the 99% of users, then the fixed choices have the following advantages:

It is easier to make a decision when you have fewer choices. If you can enter any number of minutes, then you have to stop and think – do I want 10, 15, 20 or 30 here? And how many minutes are in a day again?
Easier to make a selection – it is a single click instead of: focusing a field, clearing its value, typing a number

Do you have any plans on implementing matrix recurring notifications?

I don't have concrete plans about anything here yet, just brainstorming, but in theory, yes. Different integration types would need different notification templates – we cannot send a blob of formatted HTML and CSS in a Matrix message like we can in email, for example.

@cuu508 commented on GitHub (Aug 10, 2022): > Is there a reason we couldn't set the reminder timer to X minutes, instead of it being hardcoded 10/60/1440? If we can pick 2-4 choices that would satisfy the 99% of users, then the fixed choices have the following advantages: * It is easier to make a decision when you have fewer choices. If you can enter any number of minutes, then you have to stop and think – do I want 10, 15, 20 or 30 here? And how many minutes are in a day again? * Easier to make a selection – it is a single click instead of: focusing a field, clearing its value, typing a number > Do you have any plans on implementing matrix recurring notifications? I don't have concrete plans about anything here yet, just brainstorming, but in theory, yes. Different integration types would need different notification templates – we cannot send a blob of formatted HTML and CSS in a Matrix message like we can in email, for example.

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@rwjack commented on GitHub (Aug 10, 2022):

I don't think there needs to be a different template other than the current one. The text doesn't need to change on every repeated notification, it can be the same as the original notification. Maybe the easiest implementation would be:

after the repeat timer runs out, if the check is not up, resend enabled notification types for job (eg. email and matrix in this case)
in another case, just notify matrix

Or if you want to make it pretty and complex, global (preferably individual) checks could even have a timeout of eg. 3 notifications, and after the first notification, the second one could have appended text along the lines: "(2/3)", where 2 is the 2nd try and 3 is total notification retries. This can further be expanded into:

if 3/3 do some special action

@rwjack commented on GitHub (Aug 10, 2022): I don't think there needs to be a different template other than the current one. The text doesn't need to change on every repeated notification, it can be the same as the original notification. Maybe the easiest implementation would be: - after the repeat timer runs out, if the check is not up, resend enabled notification types for job (eg. email and matrix in this case) ![image](https://user-images.githubusercontent.com/59068073/183947946-a1a455a7-14d3-4fe1-9c76-49065905004e.png) - in another case, just notify matrix ![image](https://user-images.githubusercontent.com/59068073/183948333-44f43f73-7558-445a-92d7-1dbc581c4b8a.png) Or if you want to make it pretty and complex, global (preferably individual) checks could even have a timeout of eg. 3 notifications, and after the first notification, the second one could have appended text along the lines: "(2/3)", where 2 is the 2nd try and 3 is total notification retries. This can further be expanded into: - if 3/3 do some special action

kerem commented

2026-02-25 23:42:14 +03:00

Author

Owner

@cuu508 commented on GitHub (Jan 12, 2023):

Repeat notifications and escalation policies (if 3/3 do some special action) are best handled with dedicated incident management systems (OpsGenie, PagerDuty, Splunk On-Call, others). I'd like to keep Healthchecks focused on one task: notify when a client does not check in at the expected time. The Healthchecks notification can feed in a different system which then handles repeats, acknowledgements, snoozes, escalations, on-call schedules, issue resolutions etc.

@cuu508 commented on GitHub (Jan 12, 2023): Repeat notifications and escalation policies (if 3/3 do some special action) are best handled with dedicated incident management systems (OpsGenie, PagerDuty, Splunk On-Call, others). I'd like to keep Healthchecks focused on one task: notify when a client does not check in at the expected time. The Healthchecks notification can feed in a different system which then handles repeats, acknowledgements, snoozes, escalations, on-call schedules, issue resolutions etc.

Rows
Columns

[GH-ISSUE #510] Keep alerting when a check is down #373