starred/healthchecks

Fork 0

mirror of https://github.com/healthchecks/healthchecks.git synced 2026-04-25 23:15:49 +03:00

[GH-ISSUE #656] [Feature Request] Configurable log levels #472

New issue

Closed

opened 2026-02-25 23:42:35 +03:00 by kerem · 7 comments

kerem commented

2026-02-25 23:42:35 +03:00

Owner

Originally created by @RaphMad on GitHub (May 27, 2022).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/656

With lots of high-frequency health checks configured, the log output can become rather noisy.

One source is the "access log" which is not only triggered for visiting the dashboard manually, but also for each incoming ping:

[pid: 12|app: 0|req: 350/1399] a.b.c.d () {38 vars in 607 bytes} [Fri May 27 10:18:00 2022] GET /ping/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx => generated 2 bytes in 25 msecs (HTTP/1.1 200) 12 headers in 426 bytes (1 switches on core 0)

A second source are the regular "-- mark" messages. According to the corresponding source code comment they are meant as validation internal processes are still alive, but it would be nice being able to opt out of those as well:

-- MARK 2022-05-27T08:20:23.110771+00:00 --

Suggestion

Add settings to opt out of the mentioned log lines (default can be enabled).

Could be a single setting LOG_LEVEL or similar, or multiple ones specifying the purpose (e.g. DISABLE_ACCESSLOG, DISABLE_PERIODIC_LOG.

Originally created by @RaphMad on GitHub (May 27, 2022). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/656 With lots of high-frequency health checks configured, the log output can become rather noisy. One source is the "access log" which is not only triggered for visiting the dashboard manually, but also for each incoming ping: ```[pid: 12|app: 0|req: 350/1399] a.b.c.d () {38 vars in 607 bytes} [Fri May 27 10:18:00 2022] GET /ping/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx => generated 2 bytes in 25 msecs (HTTP/1.1 200) 12 headers in 426 bytes (1 switches on core 0)``` A second source are the regular "-- mark" messages. According to the corresponding [source code comment](https://github.com/healthchecks/healthchecks/blob/master/hc/api/management/commands/sendreports.py#L121) they are meant as validation internal processes are still alive, but it would be nice being able to opt out of those as well: ```-- MARK 2022-05-27T08:20:23.110771+00:00 --``` ## Suggestion Add settings to opt out of the mentioned log lines (default can be enabled). Could be a single setting `LOG_LEVEL` or similar, or multiple ones specifying the purpose (e.g. `DISABLE_ACCESSLOG`, `DISABLE_PERIODIC_LOG`.

kerem closed this issue

2026-02-25 23:42:35 +03:00

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 27, 2022):

What log messages would you be primarily interested in?

How would you check up on them – manually review log files occasionally?

@cuu508 commented on GitHub (May 27, 2022): What log messages would you be primarily interested in? How would you check up on them – manually review log files occasionally?

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@RaphMad commented on GitHub (May 27, 2022):

How would you check up on them – manually review log files occasionally?

For my specific use-case, I'm running via docker so the main processes stdout constitutes the log source.
The compose stacks are managed as systemd units, so all that logging lands in the systemd journal.

And then finally the systemd journal of that machine is ingested into Loki, which can be used to conveniently review the logs and could even define patterns on which to alert.

Also note that healthchecks sits behind a reverse proxy, which already performs access logging so the access log lines are actually redundant in my environment.

What log messages would you be primarily interested in?

The question is hard to answer in this direction, I know that I don't want the mentioned log lines since they are either redundant (Access log), or provide little information at the cost of clutter (-- Mark).

@RaphMad commented on GitHub (May 27, 2022): _How would you check up on them – manually review log files occasionally?_ For my specific use-case, I'm running via docker so the main processes `stdout` constitutes the log source. The compose stacks are managed as `systemd units`, so all that logging lands in the `systemd journal`. And then finally the `systemd journal` of that machine is ingested into [Loki](https://grafana.com/oss/loki/), which can be used to conveniently review the logs and could even define patterns on which to alert. Also note that healthchecks sits behind a reverse proxy, which already performs access logging so the access log lines are actually redundant in my environment. _What log messages would you be primarily interested in?_ The question is hard to answer in this direction, I know that I don't want the mentioned log lines since they are either redundant (Access log), or provide little information at the cost of clutter (-- Mark).

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 27, 2022):

When I was implementing -- MARK -- messages, I had two goals in mind:

provide an evidence in the logs the command is still alive. Otherwise, when you look at an empty log file – how do you tell if it is empty because there have not been any alerts to send, or because the command has not been running?
if logging directly to a file, provide timestamps, so the operator can tell the approx. age of log messages

In my experience, the -- MARK -- messages have not turned out to be that useful. I don't remember cases when they helped me understand an issue or solve a mystery. So I'd be OK with removing them altogether.

The access log messages come from uwsgi. You can turn it off by adding in uwsgi.ini:

disable-logging

@cuu508 commented on GitHub (May 27, 2022): When I was implementing `-- MARK --` messages, I had two goals in mind: * provide an evidence in the logs the command is still alive. Otherwise, when you look at an empty log file – how do you tell if it is empty because there have not been any alerts to send, or because the command has not been running? * if logging directly to a file, provide timestamps, so the operator can tell the approx. age of log messages In my experience, the `-- MARK --` messages have not turned out to be that useful. I don't remember cases when they helped me understand an issue or solve a mystery. So I'd be OK with removing them altogether. The access log messages come from uwsgi. You can turn it off by adding in `uwsgi.ini`: disable-logging

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@RaphMad commented on GitHub (May 27, 2022):

Externally verifying the "lifeliness" of internal ongoing processes is a general topic that could be tackled on a broader scope (e.g. introduce a /health endpoint providing with last run timestamps, but just throwing around ideas here...).
Using periodic log-messages to achieve that seems like a bad practice (at least anywhere above TRACE/DEBUG log levels, if those would be a thing).
So if logging configuration will not become a thing, I'd also lean towards removing them.

The disable-logging setting seems useful for now, but is a bit clunky to modify when not exposed via "standard" configuration means (ENV var) in the context of the docker image. (would need to over-mount a custom uwsg.ini, which makes configuration a bit more divided. between different sources...).

@RaphMad commented on GitHub (May 27, 2022): Externally verifying the "lifeliness" of internal ongoing processes is a general topic that could be tackled on a broader scope (e.g. introduce a /health endpoint providing with last run timestamps, but just throwing around ideas here...). Using periodic log-messages to achieve that seems like a bad practice (at least anywhere above `TRACE`/`DEBUG` log levels, if those would be a thing). So if logging configuration will not become a thing, I'd also lean towards removing them. The `disable-logging` setting seems useful for now, but is a bit clunky to modify when not exposed via "standard" configuration means (ENV var) in the context of the docker image. (would need to over-mount a custom `uwsg.ini`, which makes configuration a bit more divided. between different sources...).

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 27, 2022):

uwsgi can be configured through environment variables: https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#environment-variables

Try setting the following environment variable:

UWSGI_DISABLE_LOGGING=1

@cuu508 commented on GitHub (May 27, 2022): uwsgi can be configured through environment variables: https://uwsgi-docs.readthedocs.io/en/latest/Configuration.html#environment-variables Try setting the following environment variable: UWSGI_DISABLE_LOGGING=1

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@RaphMad commented on GitHub (May 27, 2022):

Works!

So the disable access log configuration already indirectly exists via that setting, I guess the only thing left for that part is to have some mention of the UWSGI env vars at https://healthchecks.io/docs/self_hosted_configuration/.

@RaphMad commented on GitHub (May 27, 2022): Works! So the `disable access log` configuration already indirectly exists via that setting, I guess the only thing left for that part is to have some mention of the UWSGI env vars at https://healthchecks.io/docs/self_hosted_configuration/.

kerem commented

2026-02-25 23:42:36 +03:00

Author

Owner

@cuu508 commented on GitHub (May 27, 2022):

Good point, updated documentation:

@cuu508 commented on GitHub (May 27, 2022): Good point, updated documentation: * https://healthchecks.io/docs/self_hosted_docker/ * https://github.com/healthchecks/healthchecks/tree/master/docker