[GH-ISSUE #1141] Anyone running HC with rootless Podman? Issue with 100% CPU load

kerem commented

2026-02-25 23:43:36 +03:00

Owner

Originally created by @nurunet on GitHub (Mar 29, 2025).
Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/1141

Hi,

I am running a tiny VPS instance (1 V-CPU, 1GB RAM) for Ntfy, and decided I wanted to try out Healthchecks. For security reasons, and for my education, I am running rootless Podman under Ubuntu Server instead of Docker. So I am aware that this may behave differently from what is expected.

I am running the container as a systemd service. It seems to work fine, until - after a few minutes, hours or days, a delay I cannot reproduce - the server becomes completely unresponsive. Once I was still connected via SSH and could see that CPU load was constantly at 100% (normally around 1%), and memory was full. The logs show uWSGI processes being killed as OOM.

Is anyone else running Healthchecks with Podman? Could you share your configs? My healthchecks.container looks like this:

[Unit]
Description=Healthchecks - Open-source cron job and background task monitoring
Requires=caddy.socket caddy.service
After=caddy.socket caddy.service

[Container]
## General
ContainerName=healthchecks.example.com
# AutoUpdate=registry allows me to update the container with podman auto-update when the systemd service is running.
AutoUpdate=registry
Image=docker.io/healthchecks/healthchecks:latest
Network=mynet.network
PublishPort=2520:2525
PublishPort=8000:8000
#UserNS=keep-id:uid=1001,gid=1001

## Volumes
#Volume=/Podman/healthchecks/data:/data:U
Volume=healthchecks-volume.volume:/data

## Environment Variables
EnvironmentFile=/Podman/healthchecks/hcenv.env

[Service]
Restart=always

#[Install]
#WantedBy=default.target

The mynet.network file is very short:

[Network]

The environment variables in hcenv.env are like this:

## Environment Variables
ADMINS=healthchecks@example.com
ALLOWED_HOSTS=healthchecks.example.com,localhost,127.0.0.1
DB=sqlite
DB_NAME=/data/hc.sqlite
DEBUG=False
DEFAULT_FROM_EMAIL=healthchecks@example.com
EMAIL_HOST=mail.infomaniak.com
EMAIL_HOST_PASSWORD=<password>
EMAIL_HOST_USER=healthchecks@example.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
PING_EMAIL_DOMAIN=healthchecks.example.com
REGISTRATION_OPEN=False
SECRET_KEY=<key>
SMTPD_PORT=2525
SITE_NAME=VPS-HC
SITE_ROOT=https://healthchecks.example.com
INTEGRATIONS_ALLOW_PRIVATE_IPS=True

Any input/experience is welcome.

Originally created by @nurunet on GitHub (Mar 29, 2025). Original GitHub issue: https://github.com/healthchecks/healthchecks/issues/1141 Hi, I am running a tiny VPS instance (1 V-CPU, 1GB RAM) for Ntfy, and decided I wanted to try out Healthchecks. For security reasons, and for my education, I am running rootless Podman under Ubuntu Server instead of Docker. So I am aware that this may behave differently from what is expected. I am running the container as a systemd service. It seems to work fine, until - after a few minutes, hours or days, a delay I cannot reproduce - the server becomes completely unresponsive. Once I was still connected via SSH and could see that CPU load was constantly at 100% (normally around 1%), and memory was full. The logs show uWSGI processes being killed as OOM. Is anyone else running Healthchecks with Podman? Could you share your configs? My **healthchecks.container** looks like this: ``` [Unit] Description=Healthchecks - Open-source cron job and background task monitoring Requires=caddy.socket caddy.service After=caddy.socket caddy.service [Container] ## General ContainerName=healthchecks.example.com # AutoUpdate=registry allows me to update the container with podman auto-update when the systemd service is running. AutoUpdate=registry Image=docker.io/healthchecks/healthchecks:latest Network=mynet.network PublishPort=2520:2525 PublishPort=8000:8000 #UserNS=keep-id:uid=1001,gid=1001 ## Volumes #Volume=/Podman/healthchecks/data:/data:U Volume=healthchecks-volume.volume:/data ## Environment Variables EnvironmentFile=/Podman/healthchecks/hcenv.env [Service] Restart=always #[Install] #WantedBy=default.target ``` The **mynet.network** file is very short: `[Network]` The environment variables in **hcenv.env** are like this: ``` ## Environment Variables ADMINS=healthchecks@example.com ALLOWED_HOSTS=healthchecks.example.com,localhost,127.0.0.1 DB=sqlite DB_NAME=/data/hc.sqlite DEBUG=False DEFAULT_FROM_EMAIL=healthchecks@example.com EMAIL_HOST=mail.infomaniak.com EMAIL_HOST_PASSWORD=<password> EMAIL_HOST_USER=healthchecks@example.com EMAIL_PORT=587 EMAIL_USE_TLS=True PING_EMAIL_DOMAIN=healthchecks.example.com REGISTRATION_OPEN=False SECRET_KEY=<key> SMTPD_PORT=2525 SITE_NAME=VPS-HC SITE_ROOT=https://healthchecks.example.com INTEGRATIONS_ALLOW_PRIVATE_IPS=True ``` Any input/experience is welcome.

kerem closed this issue

2026-02-25 23:43:36 +03:00

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@cuu508 commented on GitHub (Mar 31, 2025):

Perhaps the 100% CPU usage comes from the uWSGI processes being killed and recreated over and over again?
How much free memory is there when the Healthchecks container is not running? I.e., how much memory does it have to work with? How much swap space is there?

On my system, if I run Healthchecks with uWSGI (not inside container), each uWSGI process seems to use ~70MiB of RAM. The Docker image is configured to run 4 worker processes and two background processes, so it would need 420MiB of free RAM as the absolute minimum.

@cuu508 commented on GitHub (Mar 31, 2025): Perhaps the 100% CPU usage comes from the uWSGI processes being killed and recreated over and over again? How much free memory is there when the Healthchecks container is *not* running? I.e., how much memory does it have to work with? How much swap space is there? On my system, if I run Healthchecks with uWSGI (not inside container), each uWSGI process seems to use ~70MiB of RAM. The Docker image is configured to run 4 worker processes and two background processes, so it would need 420MiB of free RAM as the absolute minimum.

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@nurunet commented on GitHub (Mar 31, 2025):

I guess you must be right. There were maybe 300MB free before starting Healthchecks. Can I limit the number of processes to test - maybe via an environment variable?

(Oh, and would there be any detrimental effects? I currently run the instance only to alert me if my daily backup job did not send a success email.)

@nurunet commented on GitHub (Mar 31, 2025): I guess you must be right. There were maybe 300MB free before starting Healthchecks. Can I limit the number of processes to test - maybe via an environment variable? (Oh, and would there be any detrimental effects? I currently run the instance only to alert me if my daily backup job did not send a success email.)

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@cuu508 commented on GitHub (Apr 1, 2025):

Can I limit the number of processes to test - maybe via an environment variable?

Yes, set UWSGI_PROCESSES=<number>

https://github.com/healthchecks/healthchecks/tree/master/docker#uwsgi-configuration

Oh, and would there be any detrimental effects

It limits the number of requests that can be handled concurrently. On a low traffic instance there would be no problem running with, say, 2 workers.

@cuu508 commented on GitHub (Apr 1, 2025): > Can I limit the number of processes to test - maybe via an environment variable? Yes, set `UWSGI_PROCESSES=<number>` https://github.com/healthchecks/healthchecks/tree/master/docker#uwsgi-configuration > Oh, and would there be any detrimental effects It limits the number of requests that can be handled concurrently. On a low traffic instance there would be no problem running with, say, 2 workers.

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@nurunet commented on GitHub (Apr 11, 2025):

I limited it to two processes, and that massively improved the RAM situation. However, when I ran apt update afterwards, something went wrong, the server got to 100% CPU load, stuck there and after a while I rebooted it. Then I added this to my .container file:

[Service]
CPUAccounting=true
CPUQuota=50%

Since then, it has been running without hiccups, as far as I can tell. Again, thanks a lot for your help.

@nurunet commented on GitHub (Apr 11, 2025): I limited it to two processes, and that massively improved the RAM situation. However, when I ran apt update afterwards, something went wrong, the server got to 100% CPU load, stuck there and after a while I rebooted it. Then I added this to my .container file: ``` [Service] CPUAccounting=true CPUQuota=50% ``` Since then, it has been running without hiccups, as far as I can tell. Again, thanks a lot for your help.

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@nurunet commented on GitHub (Apr 27, 2025):

It happened again twice this week, the VPS became completely unresponsive:

I am not sure what else to do, so maybe I will try the hosted version instead.

@nurunet commented on GitHub (Apr 27, 2025): It happened again twice this week, the VPS became completely unresponsive: # ![Image](https://github.com/user-attachments/assets/f4f1e28f-6617-452d-99a9-5c24a6426e41) I am not sure what else to do, so maybe I will try the hosted version instead.

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@cuu508 commented on GitHub (Apr 29, 2025):

I am not sure what else to do

Does the system have swap? If not, add 1GB swap.

I'm not an expert on this but going by this, in the "Out of memory" messages the "anon-rss" field shows how much RAM each process was really using.

If that's correct, the killed processes were using 80-100MB each.

If Healthchecks is not running, how much free RAM is available? To run with two uWSGI processes I'd say there should be at least 400MB free – 100MB x 2 for the web workers, 100MB for the sendalerts process, and some safety margin.

If you are feeling adventurous, you could experiment with configuring uWSGI to use one process and multiple threads –

processes=1
threads=4

But I have not tested this, and don't know if there would be any concurrency issues or not.

@cuu508 commented on GitHub (Apr 29, 2025): > I am not sure what else to do Does the system have swap? If not, add 1GB swap. --- I'm not an expert on this but [going by this](https://stackoverflow.com/questions/18845857/what-does-anon-rss-and-total-vm-mean), in the "Out of memory" messages the "anon-rss" field shows how much RAM each process was really using. If that's correct, the killed processes were using 80-100MB each. If Healthchecks is not running, how much free RAM is available? To run with two uWSGI processes I'd say there should be at least 400MB free – 100MB x 2 for the web workers, 100MB for the sendalerts process, and some safety margin. --- If you are feeling adventurous, you could experiment with configuring uWSGI to use one process and multiple threads – ``` processes=1 threads=4 ``` But I have not tested this, and don't know if there would be any concurrency issues or not.

kerem commented

2026-02-25 23:43:36 +03:00

Author

Owner

@nurunet commented on GitHub (Apr 29, 2025):

Weirdly, without Healthchecks running, there seems to be "a lot" (well, relatively) of free RAM available. If I read htop correctly, about 500MB are free. With Healthchecks running (as long as everything is well), there are still 200-300MB free.

Not sure what happens to change this, but also why the system is completely locks up.

I don't think swap is active, I'll try it.

@nurunet commented on GitHub (Apr 29, 2025): Weirdly, without Healthchecks running, there seems to be "a lot" (well, relatively) of free RAM available. If I read htop correctly, about 500MB are free. With Healthchecks running (as long as everything is well), there are still 200-300MB free. Not sure what happens to change this, but also why the system is completely locks up. I don't think swap is active, I'll try it.

Rows
Columns

[GH-ISSUE #1141] Anyone running HC with rootless Podman? Issue with 100% CPU load #790