[GH-ISSUE #3788] "Bad Gateway" after adding a proxy host and a certificate #2494

Open
opened 2026-02-26 07:35:46 +03:00 by kerem · 14 comments
Owner

Originally created by @guillaumesoucy94 on GitHub (Jun 2, 2024).
Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3788

Checklist

  • Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
    • No
  • Are you sure you're not using someone else's docker image?
    • No
  • Have you searched for similar issues (both open and closed)?
    • Yes for two hours, finds posts but could not fix my issue.

Describe the bug

"Bad Gateway" when trying to login. Was able to login and add an proxy host and when I add the certificate to the host (certificate obtained but with a time out error" every hosts entries get gone and when I log off and try to log back in, it gave me that "bad gateway" error.

The proxy is still working, just can't login and make changes.

Nginx Proxy Manager Version

v2.11.2

To Reproduce
Steps to reproduce the behavior:
I go to my NPM instance using its IP address like the usual then I try to login.

Expected behavior

The management page as usual.

Screenshots

Screenshot at 2024-06-02 13-22-59_1

Operating System

Ubuntu Server 20.04 LTS (Upgraded to 22.04 LTS to see if it would fix it but no).

Additional context

Originally created by @guillaumesoucy94 on GitHub (Jun 2, 2024). Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3788 <!-- Are you in the right place? - If you are looking for support on how to get your upstream server forwarding, please consider asking the community on Reddit. - If you are writing code changes to contribute and need to ask about the internals of the software, Gitter is the best place to ask. - If you think you found a bug with NPM (not Nginx, or your upstream server or MySql) then you are in the *right place.* --> **Checklist** - Have you pulled and found the error with `jc21/nginx-proxy-manager:latest` docker image? - No - Are you sure you're not using someone else's docker image? - No - Have you searched for similar issues (both open and closed)? - Yes for two hours, finds posts but could not fix my issue. **Describe the bug** <!-- A clear and concise description of what the bug is. --> "Bad Gateway" when trying to login. Was able to login and add an proxy host and when I add the certificate to the host (certificate obtained but with a time out error" every hosts entries get gone and when I log off and try to log back in, it gave me that "bad gateway" error. The proxy is still working, just can't login and make changes. **Nginx Proxy Manager Version** <!-- What version of Nginx Proxy Manager is reported on the login page? --> v2.11.2 **To Reproduce** Steps to reproduce the behavior: I go to my NPM instance using its IP address like the usual then I try to login. **Expected behavior** <!-- A clear and concise description of what you expected to happen. --> The management page as usual. **Screenshots** <!-- If applicable, add screenshots to help explain your problem. --> ![Screenshot at 2024-06-02 13-22-59_1](https://github.com/NginxProxyManager/nginx-proxy-manager/assets/22561876/61e3bbff-b98c-4072-b1a7-2a6fd932e852) **Operating System** <!-- Please specify if using a Rpi, Mac, orchestration tool or any other setups that might affect the reproduction of this error. --> Ubuntu Server 20.04 LTS (Upgraded to 22.04 LTS to see if it would fix it but no). **Additional context** <!-- Add any other context about the problem here, docker version, browser version, logs if applicable to the problem. Too much info is better than too little. -->
Author
Owner

@guillaumesoucy94 commented on GitHub (Jun 5, 2024):

Yeah this what I thought. Things breaks right after attempting to create a certificate.

I forgot that the sub domain wasn't created yet on my registrar before requesting the certificate.

Do you know now how I can fix this mess?

Thanks,

Guillaume

<!-- gh-comment-id:2151127915 --> @guillaumesoucy94 commented on GitHub (Jun 5, 2024): Yeah this what I thought. Things breaks right after attempting to create a certificate. I forgot that the sub domain wasn't created yet on my registrar before requesting the certificate. Do you know now how I can fix this mess? Thanks, Guillaume
Author
Owner

@guillaumesoucy94 commented on GitHub (Jul 3, 2024):

What happened is that I requested the certificate without having created the dns entry for the subdomain at first.

So, it fails to request it, possibility because of that.

I realized the mistake, I quickly proceeded with the DNS entry creation from my domain registration interface, but it didn't fix the issue.

Guillaume

Edit

If someone from the dev team can please also give a look at the issue.

This morning, I noticed that some of my certificates just expired. Affected websites and hosted apps are now throwing certificate issues.

And also @LukeHoneyball thanks for at least getting a look into the issue. I really appreciate it 🙏

<!-- gh-comment-id:2206966096 --> @guillaumesoucy94 commented on GitHub (Jul 3, 2024): What happened is that I requested the certificate without having created the dns entry for the subdomain at first. So, it fails to request it, possibility because of that. I realized the mistake, I quickly proceeded with the DNS entry creation from my domain registration interface, but it didn't fix the issue. Guillaume *Edit* If someone from the dev team can please also give a look at the issue. This morning, I noticed that some of my certificates just expired. Affected websites and hosted apps are now throwing certificate issues. And also @LukeHoneyball thanks for at least getting a look into the issue. I really appreciate it 🙏
Author
Owner

@guillaumesoucy94 commented on GitHub (Jul 9, 2024):

Hello,

I still need help with the issue. It's still saying "bad gateway" when trying to login.

Thanks,

Guillaume

<!-- gh-comment-id:2218706148 --> @guillaumesoucy94 commented on GitHub (Jul 9, 2024): Hello, I still need help with the issue. It's still saying "bad gateway" when trying to login. Thanks, Guillaume
Author
Owner

@guillaumesoucy94 commented on GitHub (Jul 11, 2024):

fallback_error.log file is full of lines like:

2024/07/11 22:20:26 [error] 347#347: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.2.100, server: nginxproxymanager, request: "POST /api/tokens HTTP/1.1", upstream: "http://127.0.0.1:3000/tokens", host: "192.168.2.136:81", referrer: "http://192.168.2.136:81/login"

<!-- gh-comment-id:2224053437 --> @guillaumesoucy94 commented on GitHub (Jul 11, 2024): fallback_error.log file is full of lines like: `2024/07/11 22:20:26 [error] 347#347: *109 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.2.100, server: nginxproxymanager, request: "POST /api/tokens HTTP/1.1", upstream: "http://127.0.0.1:3000/tokens", host: "192.168.2.136:81", referrer: "http://192.168.2.136:81/login"`
Author
Owner

@guillaumesoucy94 commented on GitHub (Jul 18, 2024):

Any ideas? Please? A lot of my sites aren't working properly now...

<!-- gh-comment-id:2236590012 --> @guillaumesoucy94 commented on GitHub (Jul 18, 2024): Any ideas? Please? A lot of my sites aren't working properly now...
Author
Owner

@tinetoalves commented on GitHub (Jul 22, 2024):

Hello, I have the same issue happening in my environment. If I stack a new npm, I can access and configure all services/sites. However, when I issued a new certificate for a site, if I logout the admin page, I cannot access again and get the 'bad gateway' with the identical message on fallback_error.log.

<!-- gh-comment-id:2242672992 --> @tinetoalves commented on GitHub (Jul 22, 2024): Hello, I have the same issue happening in my environment. If I stack a new npm, I can access and configure all services/sites. However, when I issued a new certificate for a site, if I logout the admin page, I cannot access again and get the 'bad gateway' with the identical message on fallback_error.log.
Author
Owner

@riruigit commented on GitHub (Aug 6, 2024):

me too , and i not good idea to deal with

<!-- gh-comment-id:2271659796 --> @riruigit commented on GitHub (Aug 6, 2024): me too , and i not good idea to deal with
Author
Owner

@guillaumesoucy94 commented on GitHub (Aug 8, 2024):

Hello,

Just want to add that I will soon running NPM from a physical machine rather than on a VM, for capacity related reasons. I will also need to know how to migrate every hosts to another instance of NPM. It is something possible to do? Or, how to obtain a list of every hosts so I will be able to recreate them manually? As I cannot remember every of them.

Regards,

Guillaume

<!-- gh-comment-id:2276293312 --> @guillaumesoucy94 commented on GitHub (Aug 8, 2024): Hello, Just want to add that I will soon running NPM from a physical machine rather than on a VM, for capacity related reasons. I will also need to know how to migrate every hosts to another instance of NPM. It is something possible to do? Or, how to obtain a list of every hosts so I will be able to recreate them manually? As I cannot remember every of them. Regards, Guillaume
Author
Owner

@discokugel commented on GitHub (Oct 8, 2024):

I have the same problem, acme certificate did not recreate, no login into the ui (Bad Gateway) possible.

What helped for the recreation was:

https://github.com/NginxProxyManager/nginx-proxy-manager/issues/1499

docker exec -it npm-db-1 /bin/sh
cd /var/lib/mysql
chown -R mysql:mysql npm
exit

But after a reboot the same problem occured- but sites worg (till next recreation)

<!-- gh-comment-id:2400349463 --> @discokugel commented on GitHub (Oct 8, 2024): I have the same problem, acme certificate did not recreate, no login into the ui (Bad Gateway) possible. What helped for the recreation was: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/1499 > docker exec -it npm-db-1 /bin/sh > cd /var/lib/mysql > chown -R mysql:mysql npm > exit > But after a reboot the same problem occured- but sites worg (till next recreation)
Author
Owner

@guillaumesoucy94 commented on GitHub (Dec 2, 2024):

I suspect it may related to the database who is not connected any more to NPM.

MySQL is running:
`service mysql status
● mysql.service - MySQL Community Server
Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2024-11-30 08:41:09 EST; 2 days ago
Process: 45157 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
Main PID: 45165 (mysqld)
Status: "Server is operational"
Tasks: 39 (limit: 2219)
Memory: 341.8M
CPU: 16min 2.524s
CGroup: /system.slice/mysql.service
└─45165 /usr/sbin/mysqld

Nov 30 08:40:58 proxy-dc systemd[1]: Starting MySQL Community Server...
Nov 30 08:41:09 proxy-dc systemd[1]: Started MySQL Community Server.`

However, the docker-compose.yml look like this:

version: '3' services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' environment: DB_MYSQL_HOST: "localhost" DB_MYSQL_PORT: 3306 DB_MYSQL_USER: "npm" DB_MYSQL_PASSWORD: "npm" DB_MYSQL_NAME: "npm" volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt db: image: 'jc21/mariadb-aria:latest' restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: 'npm' MYSQL_DATABASE: 'npm' MYSQL_USER: 'npm' MYSQL_PASSWORD: 'npm' volumes: - ./data/mysql:/var/lib/mysql

The database npm was not existing when doing:

SHOW DATABASES;

I recreate it manually and:

GRANT ALL PRIVILEGES ON *.* TO 'npm'@'localhost' WITH GRANT OPTION;

so this: SHOW GRANTS FOR 'npm'@'localhost'; returns me that: +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Grants for npm@localhost | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO npm@localhostWITH GRANT OPTION | | GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ABORT_EXEMPT,AUDIT_ADMIN,AUTHENTICATION_POLICY_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FIREWALL_EXEMPT,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,GROUP_REPLICATION_STREAM,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PASSWORDLESS_USER_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SENSITIVE_VARIABLES_OBSERVER,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,TELEMETRY_LOG_ADMIN,XA_RECOVER_ADMIN ON *.* TOnpm@localhost WITH GRANT OPTION | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2 rows in set (0.00 sec)

I don't know what to do next...

Guillaume

<!-- gh-comment-id:2512976457 --> @guillaumesoucy94 commented on GitHub (Dec 2, 2024): I suspect it may related to the database who is not connected any more to NPM. MySQL is running: `service mysql status ● mysql.service - MySQL Community Server Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2024-11-30 08:41:09 EST; 2 days ago Process: 45157 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS) Main PID: 45165 (mysqld) Status: "Server is operational" Tasks: 39 (limit: 2219) Memory: 341.8M CPU: 16min 2.524s CGroup: /system.slice/mysql.service └─45165 /usr/sbin/mysqld Nov 30 08:40:58 proxy-dc systemd[1]: Starting MySQL Community Server... Nov 30 08:41:09 proxy-dc systemd[1]: Started MySQL Community Server.` However, the docker-compose.yml look like this: `version: '3' services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' environment: DB_MYSQL_HOST: "localhost" DB_MYSQL_PORT: 3306 DB_MYSQL_USER: "npm" DB_MYSQL_PASSWORD: "npm" DB_MYSQL_NAME: "npm" volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt db: image: 'jc21/mariadb-aria:latest' restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: 'npm' MYSQL_DATABASE: 'npm' MYSQL_USER: 'npm' MYSQL_PASSWORD: 'npm' volumes: - ./data/mysql:/var/lib/mysql` The database npm was not existing when doing: `SHOW DATABASES;` I recreate it manually and: `GRANT ALL PRIVILEGES ON *.* TO 'npm'@'localhost' WITH GRANT OPTION;` so this: `SHOW GRANTS FOR 'npm'@'localhost';` returns me that: `+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Grants for npm@localhost | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, SHUTDOWN, PROCESS, FILE, REFERENCES, INDEX, ALTER, SHOW DATABASES, SUPER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, REPLICATION SLAVE, REPLICATION CLIENT, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, CREATE USER, EVENT, TRIGGER, CREATE TABLESPACE, CREATE ROLE, DROP ROLE ON *.* TO `npm`@`localhost` WITH GRANT OPTION | | GRANT APPLICATION_PASSWORD_ADMIN,AUDIT_ABORT_EXEMPT,AUDIT_ADMIN,AUTHENTICATION_POLICY_ADMIN,BACKUP_ADMIN,BINLOG_ADMIN,BINLOG_ENCRYPTION_ADMIN,CLONE_ADMIN,CONNECTION_ADMIN,ENCRYPTION_KEY_ADMIN,FIREWALL_EXEMPT,FLUSH_OPTIMIZER_COSTS,FLUSH_STATUS,FLUSH_TABLES,FLUSH_USER_RESOURCES,GROUP_REPLICATION_ADMIN,GROUP_REPLICATION_STREAM,INNODB_REDO_LOG_ARCHIVE,INNODB_REDO_LOG_ENABLE,PASSWORDLESS_USER_ADMIN,PERSIST_RO_VARIABLES_ADMIN,REPLICATION_APPLIER,REPLICATION_SLAVE_ADMIN,RESOURCE_GROUP_ADMIN,RESOURCE_GROUP_USER,ROLE_ADMIN,SENSITIVE_VARIABLES_OBSERVER,SERVICE_CONNECTION_ADMIN,SESSION_VARIABLES_ADMIN,SET_USER_ID,SHOW_ROUTINE,SYSTEM_USER,SYSTEM_VARIABLES_ADMIN,TABLE_ENCRYPTION_ADMIN,TELEMETRY_LOG_ADMIN,XA_RECOVER_ADMIN ON *.* TO `npm`@`localhost` WITH GRANT OPTION | +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2 rows in set (0.00 sec)` I don't know what to do next... Guillaume
Author
Owner

@guillaumesoucy94 commented on GitHub (Dec 2, 2024):

I also rebooted, the bad gateway error is still here.

Also upgraded to v2.12.1 with theses: docker compose pull
docker compose up -d without any changes in the issue.

<!-- gh-comment-id:2512990363 --> @guillaumesoucy94 commented on GitHub (Dec 2, 2024): I also rebooted, the bad gateway error is still here. Also upgraded to v2.12.1 with theses: `docker compose pull` `docker compose up -d` without any changes in the issue.
Author
Owner

@guillaumesoucy94 commented on GitHub (Jan 18, 2025):

I just noticed those two lines in fallback_error.log

2025/01/18 22:41:48 [alert] 310#310: cache manager process 339 exited on signal 9

and

2025/01/18 22:41:48 [alert] 310#310: worker process 338 exited on signal 9

It looks like if a process crashed or something. Is this could be related the the issue?

Another thing is, if I rename docker-compose.yml doing then recreating it containing the defaults values and doing a docker compose up -dI I'm able to login with the default credentials:

Image

At the moment I put back the original docker-compose.yml file and doing docker compose up -d, I'm getting 'Bad gateway' error again.

Here the content of docker-compose.yml

services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' environment: DB_MYSQL_HOST: "127.0.0.1" DB_MYSQL_PORT: 3306 DB_MYSQL_USER: "npm" DB_MYSQL_PASSWORD: "npm" DB_MYSQL_NAME: "npm" volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt db: image: 'jc21/mariadb-aria:latest' restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: 'npm' MYSQL_DATABASE: 'npm' MYSQL_USER: 'npm' MYSQL_PASSWORD: 'npm' volumes: - ./data/mysql:/var/lib/mysql

<!-- gh-comment-id:2600253641 --> @guillaumesoucy94 commented on GitHub (Jan 18, 2025): I just noticed those two lines in fallback_error.log `2025/01/18 22:41:48 [alert] 310#310: cache manager process 339 exited on signal 9` and `2025/01/18 22:41:48 [alert] 310#310: worker process 338 exited on signal 9` It looks like if a process crashed or something. Is this could be related the the issue? Another thing is, if I rename docker-compose.yml doing then recreating it containing the defaults values and doing a `docker compose up -dI` I'm able to login with the default credentials: ![Image](https://github.com/user-attachments/assets/30134657-0e54-4756-8d39-92f5a4eebdda) At the moment I put back the original docker-compose.yml file and doing `docker compose up -d`, I'm getting 'Bad gateway' error again. Here the content of docker-compose.yml `services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' environment: DB_MYSQL_HOST: "127.0.0.1" DB_MYSQL_PORT: 3306 DB_MYSQL_USER: "npm" DB_MYSQL_PASSWORD: "npm" DB_MYSQL_NAME: "npm" volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt db: image: 'jc21/mariadb-aria:latest' restart: unless-stopped environment: MYSQL_ROOT_PASSWORD: 'npm' MYSQL_DATABASE: 'npm' MYSQL_USER: 'npm' MYSQL_PASSWORD: 'npm' volumes: - ./data/mysql:/var/lib/mysql`
Author
Owner

@guillaumesoucy94 commented on GitHub (Jan 22, 2025):

@discokugel

I did tried chown -R mysql:mysql npm

The only thing that I'd changed was the docker exec -it npm-db-1 /bin/sh by docker exec -it root-app-1 /bin/sh to match my setup.

Unfortunately it still won't work.

<!-- gh-comment-id:2608351505 --> @guillaumesoucy94 commented on GitHub (Jan 22, 2025): @discokugel I did tried `chown -R mysql:mysql npm` The only thing that I'd changed was the `docker exec -it npm-db-1 /bin/sh` by `docker exec -it root-app-1 /bin/sh` to match my setup. Unfortunately it still won't work.
Author
Owner

@github-actions[bot] commented on GitHub (Jul 31, 2025):

Issue is now considered stale. If you want to keep it open, please comment 👍

<!-- gh-comment-id:3138350086 --> @github-actions[bot] commented on GitHub (Jul 31, 2025): Issue is now considered stale. If you want to keep it open, please comment :+1:
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/nginx-proxy-manager-NginxProxyManager#2494
No description provided.