[GH-ISSUE #28] 504 timeout when uploading custom certificates #27

Closed
opened 2026-02-26 05:33:08 +03:00 by kerem · 4 comments
Owner

Originally created by @SGStino on GitHub (Nov 19, 2018).
Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/28

I'm getting a classic nginx 504 Gateway Time-out error page response when adding an new custom certificate:

POST http://proxy.docker/api/nginx/certificates/validate

there are no log entries appearing during the period between submit request and timout response. However, there are 2 lines that don't look healthy:

[11/19/2018] [3:44:21 PM] [Access  ] › ✖  error                        users:get 1 Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
[11/19/2018] [3:44:21 PM] [Express ] › ⚠  warning                      Permission Denied

this also means that it's probably not issue #9 that returned?

Is there a place where I can discover more logging?

Originally created by @SGStino on GitHub (Nov 19, 2018). Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/28 I'm getting a classic nginx 504 Gateway Time-out error page response when adding an new custom certificate: `POST http://proxy.docker/api/nginx/certificates/validate` there are no log entries appearing during the period between submit request and timout response. However, there are 2 lines that don't look healthy: ``` [11/19/2018] [3:44:21 PM] [Access ] › ✖ error users:get 1 Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call? [11/19/2018] [3:44:21 PM] [Express ] › ⚠ warning Permission Denied ``` this also means that it's probably not issue #9 that returned? Is there a place where I can discover more logging?
kerem closed this issue 2026-02-26 05:33:08 +03:00
Author
Owner

@SGStino commented on GitHub (Nov 19, 2018):

Ok, Knex seems to be the connection to MariaDb?

So i looked into the maria logging and found this:
2018-11-19 15:59:13 92 [Warning] Aborted connection 92 to db: 'npm' user: 'root' host: '172.17.0.1' (Got timeout reading communication packets)

a lot of them ..

why would it abort connections while also allowing others (since nginx proxy manager seems to work for some (most) tasks)?

<!-- gh-comment-id:439950567 --> @SGStino commented on GitHub (Nov 19, 2018): Ok, Knex seems to be the connection to MariaDb? So i looked into the maria logging and found this: ```2018-11-19 15:59:13 92 [Warning] Aborted connection 92 to db: 'npm' user: 'root' host: '172.17.0.1' (Got timeout reading communication packets) ``` a lot of them .. why would it abort connections while also allowing others (since nginx proxy manager seems to work for some (most) tasks)?
Author
Owner

@jc21 commented on GitHub (Nov 26, 2018):

Hmm interesting. Not sure how the knex issues are linked to the file system writing a cert, but let's look at the knex thing first.

Timeout acquiring a connection. The pool is probably full.

Just guessing here, is the disk hosting the database files full? Is the device running the docker containers loaded up and hitting the maximum number of open files allowed?

Are you running this on a pi, synology or some other non-bare metal non pc setup?

What's your docker-compose.yml or docker run commands look like?

<!-- gh-comment-id:441515766 --> @jc21 commented on GitHub (Nov 26, 2018): Hmm interesting. Not sure how the knex issues are linked to the file system writing a cert, but let's look at the knex thing first. > Timeout acquiring a connection. The pool is probably full. Just guessing here, is the disk hosting the database files full? Is the device running the docker containers loaded up and hitting the maximum number of open files allowed? Are you running this on a pi, synology or some other non-bare metal non pc setup? What's your `docker-compose.yml` or docker run commands look like?
Author
Owner

@SGStino commented on GitHub (Nov 28, 2018):

I've created mariadb with portainer. But if i were to guess from the settings it would be
docker run -p 3306:3306 -v mariadb:/var/lib/mysql mariadb:latest

for "npm":
docker run -p 81:81 -p 80:80 -p 443:443 -v /etc/letsencrypt:/etc/letsencrypt -v /etc/nginx-proxy-manager/config.json:/app/config/production.json -v /var/nginx-proxy-manager/data:/data jc21/nginx-proxy-manager:latest

it's running on Ubuntu 18.04 on a VMWare ESX infra, so I'd guess it's not going to be hardware constraints from looking at the usage in vSphere. But there was a short period where the host ran out of diskspace for a while.

Does mariadb not recover from errors like that perhaps?

currently npm's output contains the following:

[11/28/2018] [2:47:01 PM] [Access  ] › ✖  error                        users:get 1 select `user`.* from `user` where `id` = 1 and `is_deleted` = 0 and `is_disabled` = 0 - Packets out of order. Got: 0 Expected: 1
[11/28/2018] [2:47:01 PM] [Express ] › ⚠  warning                      Permission Denied 

so a guess from someone not having any experience with mariadb:

did the temporary "out of diskspace" issue cause mariadb to generate bad records, that it's now blindly replaying to it's clients, who do check if the record packets are in order?

But I'm not entirely sure if those errors are even related to the certificates, because new attempts at uploading certificates don't generate any more log entries. Hence why I asked if I could find some more detailed logging somewhere? :) It looks more like the error is from the moment I tried to login (which did work).

EDIT:
it would seem that the timeout also occurs when adding new http proxy entries, however, those seem to be added after refreshing. If there is any way I could increase the logging verbosity I might be able to pinpoint the issue?

<!-- gh-comment-id:442475591 --> @SGStino commented on GitHub (Nov 28, 2018): I've created mariadb with portainer. But if i were to guess from the settings it would be `docker run -p 3306:3306 -v mariadb:/var/lib/mysql mariadb:latest` for "npm": `docker run -p 81:81 -p 80:80 -p 443:443 -v /etc/letsencrypt:/etc/letsencrypt -v /etc/nginx-proxy-manager/config.json:/app/config/production.json -v /var/nginx-proxy-manager/data:/data jc21/nginx-proxy-manager:latest` it's running on Ubuntu 18.04 on a VMWare ESX infra, so I'd guess it's not going to be hardware constraints from looking at the usage in vSphere. But there was a short period where the host ran out of diskspace for a while. Does mariadb not recover from errors like that perhaps? currently npm's output contains the following: ``` [11/28/2018] [2:47:01 PM] [Access ] › ✖ error users:get 1 select `user`.* from `user` where `id` = 1 and `is_deleted` = 0 and `is_disabled` = 0 - Packets out of order. Got: 0 Expected: 1 [11/28/2018] [2:47:01 PM] [Express ] › ⚠ warning Permission Denied ``` so a guess from someone not having any experience with mariadb: did the temporary "out of diskspace" issue cause mariadb to generate bad records, that it's now blindly replaying to it's clients, who do check if the record packets are in order? But I'm not entirely sure if those errors are even related to the certificates, because new attempts at uploading certificates don't generate any more log entries. Hence why I asked if I could find some more detailed logging somewhere? :) It looks more like the error is from the moment I tried to login (which did work). EDIT: it would seem that the timeout also occurs when adding new http proxy entries, however, those seem to be added after refreshing. If there is any way I could increase the logging verbosity I might be able to pinpoint the issue?
Author
Owner

@chaptergy commented on GitHub (May 10, 2021):

A lot has changed since this issue was opened, and currently the certificate upload is working. If you have any further problems please open a new issue.

<!-- gh-comment-id:837174562 --> @chaptergy commented on GitHub (May 10, 2021): A lot has changed since this issue was opened, and currently the certificate upload is working. If you have any further problems please open a new issue.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/nginx-proxy-manager-NginxProxyManager#27
No description provided.