[GH-ISSUE #3610] NPM crashes when deleting a few hosts and ssl certs, then resets to default and it is empty without any records #2390

Open
opened 2026-02-26 07:35:22 +03:00 by kerem · 9 comments
Owner

Originally created by @comfuzio on GitHub (Mar 7, 2024).
Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3610

Hi, I am running NPM for more than 1 year now in a proxmox vm from template of ubuntu 22.04.3 server (full updated).
I have installed docker via the official docker instructions and then portainer for gui.
My docker compose file is this:
`version: '3.3'

services:
app:
image: 'jc21/nginx-proxy-manager:latest'
restart: unless-stopped
ports:
- '80:80'
- '81:81'
- '443:443'
volumes:
- ./data:/data
- ./letsencrypt:/etc/letsencrypt

goaccess:
image: 'xavierh/goaccess-for-nginxproxymanager:latest'
container_name: goaccess
restart: always
ports:
- '7880:7880'
environment:
- PUID=0
- PGID=0
- TZ=Europe/Athens
- SKIP_ARCHIVED_LOGS=False # optional
- DEBUG=False # optional
- BASIC_AUTH=False # optional
- BASIC_AUTH_USERNAME=user # optional
- BASIC_AUTH_PASSWORD=pass # optional
- EXCLUDE_IPS=127.0.0.1 # optional - comma delimited
- LOG_TYPE=NPM # optional - more information below
volumes:
- ./data/logs:/opt/log
`
to clarify the problem which I will describe happens also before I have added the goaccess to my npm compose.
So I have a few domain names and some subdomains etc and it works perfect and thank you for this amazing project!
The problem happens now that I am moving my websites to another VM running a webhosting software than keeping them in docker compose.
I have migrated all my sites to the new host, change A records to match the new IP and all runs good!
So I go to NPM to delete those sites, if I delete 1-2 domains and their certs, I don't see any problem but when I try to delete more, one at a time of course, NPM is not responding.
So I stop the stack and restart it, when it starts, it is in factory default settings! I have to login with the default info and it doesn't contain any record at all!
So I roll back to the backup I am always taking before I do such actions.

Checklist

  • Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
    • Yes
  • Are you sure you're not using someone else's docker image?
    • Yes
  • Have you searched for similar issues (both open and closed)?
    • Yes

Describe the bug

NPM crashes when deleting hosts and certs and when re-started it is in factory default state.

Nginx Proxy Manager Version

v2.11.1

To Reproduce
Steps to reproduce the behavior:

  1. Go to 'Hosts>proxy hosts'
  2. Click on '3 dots on a host and delete'
  3. Go to 'SSL Certificates'
  4. Delete the matching record.
  5. Repeat that 5 times or so.
  6. NPM is not responding.
  7. Restart service and NPM is in factory defaults.

Expected behavior

Delete the hosts and certifications and not crash or lose all remaining records after restart

Operating System

Ubuntu 22.04.3 server full updated on proxmox 8.1 host

notes
I had an ARM64 vm debian 12 in hetzner with same installation of docker and portainer and npm with this yaml and it did the same there as well.
It did the same even before i discovered the goaccess

Originally created by @comfuzio on GitHub (Mar 7, 2024). Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3610 Hi, I am running NPM for more than 1 year now in a proxmox vm from template of ubuntu 22.04.3 server (full updated). I have installed docker via the official docker instructions and then portainer for gui. My docker compose file is this: `version: '3.3' services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt goaccess: image: 'xavierh/goaccess-for-nginxproxymanager:latest' container_name: goaccess restart: always ports: - '7880:7880' environment: - PUID=0 - PGID=0 - TZ=Europe/Athens - SKIP_ARCHIVED_LOGS=False # optional - DEBUG=False # optional - BASIC_AUTH=False # optional - BASIC_AUTH_USERNAME=user # optional - BASIC_AUTH_PASSWORD=pass # optional - EXCLUDE_IPS=127.0.0.1 # optional - comma delimited - LOG_TYPE=NPM # optional - more information below volumes: - ./data/logs:/opt/log ` to clarify the problem which I will describe happens also before I have added the goaccess to my npm compose. So I have a few domain names and some subdomains etc and it works perfect and thank you for this amazing project! The problem happens now that I am moving my websites to another VM running a webhosting software than keeping them in docker compose. I have migrated all my sites to the new host, change A records to match the new IP and all runs good! So I go to NPM to delete those sites, if I delete 1-2 domains and their certs, I don't see any problem but when I try to delete more, one at a time of course, NPM is not responding. So I stop the stack and restart it, when it starts, it is in factory default settings! I have to login with the default info and it doesn't contain any record at all! So I roll back to the backup I am always taking before I do such actions. **Checklist** - Have you pulled and found the error with `jc21/nginx-proxy-manager:latest` docker image? - Yes - Are you sure you're not using someone else's docker image? - Yes - Have you searched for similar issues (both open and closed)? - Yes **Describe the bug** <!-- A clear and concise description of what the bug is. --> NPM crashes when deleting hosts and certs and when re-started it is in factory default state. **Nginx Proxy Manager Version** <!-- What version of Nginx Proxy Manager is reported on the login page? --> v2.11.1 **To Reproduce** Steps to reproduce the behavior: 1. Go to 'Hosts>proxy hosts' 2. Click on '3 dots on a host and delete' 3. Go to 'SSL Certificates' 4. Delete the matching record. 5. Repeat that 5 times or so. 6. NPM is not responding. 7. Restart service and NPM is in factory defaults. **Expected behavior** <!-- A clear and concise description of what you expected to happen. --> Delete the hosts and certifications and not crash or lose all remaining records after restart **Operating System** <!-- Please specify if using a Rpi, Mac, orchestration tool or any other setups that might affect the reproduction of this error. --> Ubuntu 22.04.3 server full updated on proxmox 8.1 host **notes** I had an ARM64 vm debian 12 in hetzner with same installation of docker and portainer and npm with this yaml and it did the same there as well. It did the same even before i discovered the goaccess
Author
Owner

@bluekitedreamer commented on GitHub (Mar 9, 2024):

Were there any wildcard domains used with any of these hosts? I experienced the same thing, but only happens when an existing proxy was still dependent upon a deleted certificate. I did a full example so I could grab the logs for further debugging on this, those are below.

During SSL cert deletion the web interface indicates that the hosts will need to be updated later, which infers it would not create a catastrophic break. This results in an unrecoverable error state.

When a certificate is deleted NPM goes out and properly revokes the certificate and then proceeds to delete it from local file store

› ℹ  info      Revoking Let'sEncrypt certificates for Cert #14: *.domain.com
› ℹ  info      Command: certbot revoke --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --cert-path "/etc/letsencrypt/live/npm-14/fullchain.pem" --delete-after-revoke ; rm -f '/etc/letsencrypt/credentials/credentials-14' || true
› ⬤  debug     CMD: certbot revoke --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --cert-path "/etc/letsencrypt/live/npm-14/fullchain.pem" --delete-after-revoke 
› ⬤  debug     CMD: rm -f '/etc/letsencrypt/credentials/credentials-14' || true
› ℹ  info      Deleted all files relating to certificate npm-14.
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file)
[SSL      ] › ℹ  info      Let's Encrypt Renewal Timer initialized
[SSL      ] › ℹ  info      Renewing SSL certs expiring within 30 days ...
[IP Ranges] › ℹ  info      IP Ranges Renewal Timer initialized
[Global   ] › ℹ  info      Backend PID 155 listening on port 3000 ...
[SSL      ] › ℹ  info      Completed SSL cert renew process
❯ Starting nginx ...
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file)
❯ Starting nginx ...
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file)
❯ Starting nginx ...
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file)

This is somewhat functionally related to a different bug I reported here https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3604, where a dependency on the cert checking/validation is causing an unrecoverable error in the backend and doesn't allow the application to start up in order to remedy using the web app.

<!-- gh-comment-id:1986763959 --> @bluekitedreamer commented on GitHub (Mar 9, 2024): Were there any wildcard domains used with any of these hosts? I experienced the same thing, but only happens when an existing proxy was still dependent upon a deleted certificate. I did a full example so I could grab the logs for further debugging on this, those are below. During SSL cert deletion the web interface indicates that the hosts will need to be updated later, which infers it would not create a catastrophic break. This results in an unrecoverable error state. When a certificate is deleted NPM goes out and properly revokes the certificate and then proceeds to delete it from local file store ``` › ℹ info Revoking Let'sEncrypt certificates for Cert #14: *.domain.com › ℹ info Command: certbot revoke --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --cert-path "/etc/letsencrypt/live/npm-14/fullchain.pem" --delete-after-revoke ; rm -f '/etc/letsencrypt/credentials/credentials-14' || true › ⬤ debug CMD: certbot revoke --config "/etc/letsencrypt.ini" --work-dir "/tmp/letsencrypt-lib" --logs-dir "/tmp/letsencrypt-log" --cert-path "/etc/letsencrypt/live/npm-14/fullchain.pem" --delete-after-revoke › ⬤ debug CMD: rm -f '/etc/letsencrypt/credentials/credentials-14' || true › ℹ info Deleted all files relating to certificate npm-14. ``` ``` nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file) [SSL ] › ℹ info Let's Encrypt Renewal Timer initialized [SSL ] › ℹ info Renewing SSL certs expiring within 30 days ... [IP Ranges] › ℹ info IP Ranges Renewal Timer initialized [Global ] › ℹ info Backend PID 155 listening on port 3000 ... [SSL ] › ℹ info Completed SSL cert renew process ❯ Starting nginx ... nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file) ❯ Starting nginx ... nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file) ❯ Starting nginx ... nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-14/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/npm-14/fullchain.pem, r) error:10000080:BIO routines::no such file) ``` This is somewhat functionally related to a different bug I reported here https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3604, where a dependency on the cert checking/validation is causing an unrecoverable error in the backend and doesn't allow the application to start up in order to remedy using the web app.
Author
Owner

@bluekitedreamer commented on GitHub (Mar 9, 2024):

Work arounds

For Now:
Manually run certbot for the deleted certs, and insert the certs into DB with the same ID as the original cert and restart the container. The original cert records will still be in the db for reference, with the is_deleted column marked as the value 1. Clone the record, change the old ID and create a new record in its place with the original ID.

In the Future:
With the above being stated, take backups of the SSL cert store (not generally recommended if security controls on the backups aren't in place) and NPM database for now.

You can easily recover by restoring the files, this will get the backend and frontend in a working state again. Once it's up go to

  1. SSL Certs > the cert > and click renew
  2. Go to each affected proxy host and re assign the certs (when the renew happens the cert ID in the database won't be updated)
<!-- gh-comment-id:1986770219 --> @bluekitedreamer commented on GitHub (Mar 9, 2024): ### Work arounds **For Now:** Manually run certbot for the deleted certs, and insert the certs into DB with the same ID as the original cert and restart the container. The original cert records will still be in the db for reference, with the `is_deleted` column marked as the value `1.` Clone the record, change the old ID and create a new record in its place with the original ID. **In the Future:** With the above being stated, take backups of the SSL cert store (not generally recommended if security controls on the backups aren't in place) and NPM database for now. You can easily recover by restoring the files, this will get the backend and frontend in a working state again. Once it's up go to 1. SSL Certs > the cert > and click renew 2. Go to each affected proxy host and re assign the certs (when the renew happens the cert ID in the database won't be updated)
Author
Owner

@comfuzio commented on GitHub (Mar 9, 2024):

most of them were in the style of domain.com and www.domain.com but some where subdomains such as subdomain.domain.com.
Indeed some of the certs if I go to the certs tab are red and outdated, this is also odd because NPM never updates them there, the certs themselves if you visit the sites, they were autorenew but the certs tab didn't seem to get the update

<!-- gh-comment-id:1986782575 --> @comfuzio commented on GitHub (Mar 9, 2024): most of them were in the style of domain.com and www.domain.com but some where subdomains such as subdomain.domain.com. Indeed some of the certs if I go to the certs tab are red and outdated, this is also odd because NPM never updates them there, the certs themselves if you visit the sites, they were autorenew but the certs tab didn't seem to get the update
Author
Owner

@comfuzio commented on GitHub (Mar 9, 2024):

For now I just want to remove all those domains that I have transferred to the other vm and just keep those that remain in the same portainer instance (they still are plenty).
My workaround for now seems to be to make a record of what I want to keep and the internal ip/port and redo them all since I don't want to create another interruption to the transferred domains to change the A record back, do SSL renew, change A record again.

<!-- gh-comment-id:1986783096 --> @comfuzio commented on GitHub (Mar 9, 2024): For now I just want to remove all those domains that I have transferred to the other vm and just keep those that remain in the same portainer instance (they still are plenty). My workaround for now seems to be to make a record of what I want to keep and the internal ip/port and redo them all since I don't want to create another interruption to the transferred domains to change the A record back, do SSL renew, change A record again.
Author
Owner

@bluekitedreamer commented on GitHub (Mar 9, 2024):

For now I just want to remove all those domains that I have transferred to the other vm and just keep those that remain in the same portainer instance (they still are plenty). My workaround for now seems to be to make a record of what I want to keep and the internal ip/port and redo them all since I don't want to create another interruption to the transferred domains to change the A record back, do SSL renew, change A record again.

yeah basically my comment but in a different way. Sometimes getting out of a one-time problem manually is the easiest answer.

<!-- gh-comment-id:1986801248 --> @bluekitedreamer commented on GitHub (Mar 9, 2024): > For now I just want to remove all those domains that I have transferred to the other vm and just keep those that remain in the same portainer instance (they still are plenty). My workaround for now seems to be to make a record of what I want to keep and the internal ip/port and redo them all since I don't want to create another interruption to the transferred domains to change the A record back, do SSL renew, change A record again. yeah basically my comment but in a different way. Sometimes getting out of a one-time problem manually is the easiest answer.
Author
Owner

@github-actions[bot] commented on GitHub (Oct 28, 2024):

Issue is now considered stale. If you want to keep it open, please comment 👍

<!-- gh-comment-id:2440359518 --> @github-actions[bot] commented on GitHub (Oct 28, 2024): Issue is now considered stale. If you want to keep it open, please comment :+1:
Author
Owner

@comfuzio commented on GitHub (Oct 28, 2024):

Is there any update on this issue? if not, it is safe to close it. As an issue, for me it still remains unfortunatelly.

<!-- gh-comment-id:2440672227 --> @comfuzio commented on GitHub (Oct 28, 2024): Is there any update on this issue? if not, it is safe to close it. As an issue, for me it still remains unfortunatelly.
Author
Owner

@dominicm00 commented on GitHub (Apr 18, 2025):

Can confirm this is still happening 👍🏼, this makes NPM insta-crash on startup unless you manually edit the DB/files

<!-- gh-comment-id:2816336823 --> @dominicm00 commented on GitHub (Apr 18, 2025): Can confirm this is still happening 👍🏼, this makes NPM insta-crash on startup unless you manually edit the DB/files
Author
Owner

@github-actions[bot] commented on GitHub (Oct 31, 2025):

Issue is now considered stale. If you want to keep it open, please comment 👍

<!-- gh-comment-id:3471024110 --> @github-actions[bot] commented on GitHub (Oct 31, 2025): Issue is now considered stale. If you want to keep it open, please comment :+1:
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/nginx-proxy-manager-NginxProxyManager#2390
No description provided.