starred/healthchecks

Fork 0

mirror of https://github.com/healthchecks/healthchecks.git synced 2026-04-25 23:15:49 +03:00

[GH-ISSUE #610] [Self-hosted] How to configure healthchecks.io in a multi-instances mode #445

New issue

Closed

opened 2026-02-25 23:42:29 +03:00 by kerem · 5 comments

kerem commented

2026-02-25 23:42:29 +03:00

Owner

Originally created by @tizard34 on GitHub (Feb 22, 2022).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/610

We have deployed Healthchecks.io in a kubernetes platform. The application is running in 2 instances (load-balanced).

When a check is down (The ping /fail is sent), we are receiving 1 notification.
When a check is down because it's to late, we sometimes feel like we get 2 notifications.

Is there a specific configuration that would allow Healthchecks.io to work in multi-instance mode ?

Originally created by @tizard34 on GitHub (Feb 22, 2022). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/610 We have deployed Healthchecks.io in a kubernetes platform. The application is running in 2 instances (load-balanced). When a check is down (The ping /fail is sent), we are receiving 1 notification. When a check is down because it's to late, we sometimes feel like we get 2 notifications. Is there a specific configuration that would allow Healthchecks.io to work in multi-instance mode ?

kerem closed this issue

2026-02-25 23:42:29 +03:00

kerem commented

2026-02-25 23:42:29 +03:00

Author

Owner

@cuu508 commented on GitHub (Feb 23, 2022):

Do both instances use the same database?

@cuu508 commented on GitHub (Feb 23, 2022): Do both instances use the same database?

kerem commented

2026-02-25 23:42:29 +03:00

Author

Owner

@tizard34 commented on GitHub (Feb 23, 2022):

@cuu508 yes of course
Are you insinuating that the sending of duplicate notifications is not related to the number of instances?

@tizard34 commented on GitHub (Feb 23, 2022): @cuu508 yes of course Are you insinuating that the sending of duplicate notifications is not related to the number of instances?

kerem commented

2026-02-25 23:42:29 +03:00

Author

Owner

@cuu508 commented on GitHub (Feb 23, 2022):

Running multiple instances should not cause duplicate notifications, unless there's a bug.

When a check is down because it's to late, we sometimes feel like we get 2 notifications.

Do you see the duplicate outgoing notifications in web UI, check's details page, Log section?

The sendalerts management command writes a log of sent notifications to stdout – can you see duplicate outgoing notifications in sendalerts logs?

@cuu508 commented on GitHub (Feb 23, 2022): Running multiple instances should not cause duplicate notifications, unless there's a bug. > When a check is down because it's to late, we sometimes feel like we get 2 notifications. Do you see the duplicate outgoing notifications in web UI, check's details page, Log section? The sendalerts management command writes a log of sent notifications to stdout – can you see duplicate outgoing notifications in sendalerts logs?

kerem commented

2026-02-25 23:42:29 +03:00

Author

Owner

@tizard34 commented on GitHub (Feb 23, 2022):

Ok, I will take care of that.
For the moment, I have simply noticed the duplicate notifications in MS Teams, not in the web UI.

@tizard34 commented on GitHub (Feb 23, 2022): Ok, I will take care of that. For the moment, I have simply noticed the duplicate notifications in MS Teams, not in the web UI.

kerem commented

2026-02-25 23:42:30 +03:00

Author

Owner

@cuu508 commented on GitHub (Feb 23, 2022):

Healthchecks posts the notification by making a HTTP request to MS Teams incoming webhook address.
If a HTTP request fails, the request is retried 1 or 2 more times. The following conditions count as failures:

the remote server responds with a non-2xx status code
the request times out (timeout is 10 seconds)
connection is refused

So in theory the following scenario is possible:

Healthchecks makes a request to MS Teams
MS Teams accepts the request, posts the message in chat, but takes 15 seconds to send a HTTP response back
After 10 seconds Healthchecks concludes the request has timed out, and retries the same request again
MS Teams accepts the repeated request, posts a duplicate message

I don't know if that's what's happening here, but it's a possibility.

@cuu508 commented on GitHub (Feb 23, 2022): Healthchecks posts the notification by making a HTTP request to MS Teams incoming webhook address. If a HTTP request fails, the request is retried 1 or 2 more times. The following conditions count as failures: * the remote server responds with a non-2xx status code * the request times out (timeout is 10 seconds) * connection is refused So in theory the following scenario is possible: * Healthchecks makes a request to MS Teams * MS Teams accepts the request, posts the message in chat, but takes 15 seconds to send a HTTP response back * After 10 seconds Healthchecks concludes the request has timed out, and retries the same request again * MS Teams accepts the repeated request, posts a duplicate message I don't know if that's what's happening here, but it's a possibility.