mirror of
https://github.com/NginxProxyManager/nginx-proxy-manager.git
synced 2026-04-26 01:45:54 +03:00
[GH-ISSUE #3154] First time boot takes incredibly long due to chown -R "$PUID:$PGID" /opt/certbot #2137
Labels
No labels
awaiting feedback
bug
cannot reproduce
dns provider request
duplicate
enhancement
enhancement
enhancement
good first issue
help wanted
invalid
need more info
no certbot plugin available
product-support
pull-request
question
stale
troll
upstream issue
v2
v2
v2
v3
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/nginx-proxy-manager-NginxProxyManager#2137
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @devedse on GitHub (Aug 23, 2023).
Original GitHub issue: https://github.com/NginxProxyManager/nginx-proxy-manager/issues/3154
Checklist
jc21/nginx-proxy-manager:latestdocker image?Describe the bug
When I start the container for the first time it's taking an incredibly long due to it setting ownership:
I did a bit of investigation and found out it's due to this command:
This code was added in this commit:
github.com/NginxProxyManager/nginx-proxy-manager@05307aa253I ran this command myself inside the container and it took about 5 minutes to complete:
To me it feels a bit strange to have to execute this command on the /opt folder which isn't even mapped to my host system. So why not do this during the creation of the container rather then during boot?
Nginx Proxy Manager Version
2.10.4
To Reproduce
See above
Expected behavior
Boot quickly
Screenshots
N/A
Operating System
Docker inside a container on a Synology NAS
Additional context
@Nightreaver commented on GitHub (Sep 11, 2023):
I can confirm this. Sometimes it pretty annyoing
@panos-stavrianos commented on GitHub (Oct 9, 2023):
Any workaround at least? 3-4 minutes downtime just to restart npm is a lot, and very annoying when changing configs and have to restart to see the results.
@devedse commented on GitHub (Oct 9, 2023):
I guess you could map a volume to the root fs that persists the /opt folder
@panos-stavrianos commented on GitHub (Oct 9, 2023):
Thanks, it worked!
@DWRedShoes commented on GitHub (Nov 9, 2023):
I've ran into the exact same issue at startup of the docker container where the chown command (Setting Ownership) took around 25 minutes to complete and this happens every time I turn the container on.
@anantanandgupta commented on GitHub (Jan 4, 2024):
How? can you please explain? I am not getting pass to the settings ownership and it simply timesout in my case. I am mounting the data and letsencrypt folder on an nfs share.
@corporategoth commented on GitHub (Jan 9, 2024):
I too am hitting this issue on my TrueNAS system. How I resolve this for now.
Step 1: Create a PVC mount called /opt/certbot2 (ie. an external mount is managed by TrueNAS - this is not the same as a named mount managed by docker, from docker's perspective it's still a host mount, but it's as good as you get with truenas).
Step 2: Start the container, then log into it via. shell (use heavyscript or truenas to get into the shell)
Step 3: Copy everything in /opt/certbot to /opt/certbot2 (cp -a /opt/certbot/* /opt/certbot2)
Step 4: Stop the container and change the PVC mount from /opt/certbot2 to /opt/certbot
Step 5: Start the container again.
It should now be very quick to chown, because the storage is external, not copy on write.
Of course, this is a terrible solution, because we are now overriding the docker image for what is in /opt/certbot, so as the container updates, our /opt/certbot will not unless you remove the PVC and do the above steps again.
@Lapo-Statix commented on GitHub (Feb 19, 2024):
@panos-stavrianos
How did you do that? I'm having the same problem running on my server, when I run it on my machine it doesn't take long. 😢
@Nightreaver commented on GitHub (Feb 19, 2024):
I would guess,
run the container and wait for it to start,
use docker copy and copy the /opt from the container to a folder on your
harddisk
once this is done, stop the container
add a volume that maps the copy on your local disk to the /opt inside the
container
start the container again
Am Mo., 19. Feb. 2024 um 18:45 Uhr schrieb Lapo @.***>:
@Lapo-Statix commented on GitHub (Feb 19, 2024):
@Nightreaver
Thank you!! it worked :)
@willtaylor commented on GitHub (Feb 25, 2024):
I'm having the same problem with 2.11.1 via the TrueNAS catalog chart version 1.0.27
I never seem to get past the chown before TrueNAS decides that the container is unhealthy and attempts to restart it, causing an endless loop
I haven't tried the proposed workaround
EDIT
I was just in the console when the container was restarted and it echoed the following message to the console:
command terminated with exit code 137@dasunsrule32 commented on GitHub (Apr 9, 2024):
Affected by this issue as well on TrueNAS SCALE 23.10.2. Same issue on the community train and installing the docker image directly as well. Takes about 10 minutes for mine to fully load.
@dasunsrule32 commented on GitHub (Apr 9, 2024):
So I copied the files to my HDD pool in my
/datamount that mounts to npm and created a folder there calledcertbotand mounted a host path of/opt/certbotto/data/certbotit has the same effect. Boots in seconds now.@Furglitch commented on GitHub (Apr 11, 2024):
This actually really helped me out.
Working in Portainer, I just edited my stack compose.
First added the following
waited for the app to output the command response (a ton of lines starting with '/opt/certbot
then changed to
and restarted. npm loaded pretty much immediately, and I was able to access the admin panel.
restarted again just to make sure, and npm loaded in less than 10 seconds.
I'm still learning docker so I may not be doing it the best way, but it worked for me.
@dasunsrule32 commented on GitHub (Apr 11, 2024):
Yep, same thing I did, just using docker compose.
@MichaelKirgus commented on GitHub (Apr 13, 2024):
This cant be an solution, only an workaround. Its VERY anoying...at every update of the container or helm forces an downtime for over an hour...
@dasunsrule32 commented on GitHub (Apr 13, 2024):
Not a solution, just a workaround for now.
@MichaelKirgus commented on GitHub (Apr 13, 2024):
Ok, sorry if my comment was a bit salty...lets be productive:
Is there an elegant solution for this? What about executing chrown only if PUID/PGID is set or check if the first file under "/data/certbot" has the right permissions and owner and then skip it?
@nintendoeats commented on GitHub (Aug 21, 2024):
Did anybody ever figure out an answer to the original question of why this is required? Could we simply...remove the offending line?....
@Mathpro commented on GitHub (Sep 4, 2024):
Same issue here unfortunately
@Dremor commented on GitHub (Sep 6, 2024):
Same here, Truenas 24.10-Beta.1.
10 to 15 minutes downtime, on an SSD array. I can imagine how frustrating it must be on a pure HDD array.
@Tsaukpaetra commented on GitHub (Sep 7, 2024):
I think instead of using a hacky workaround, perhaps the underlying issue with whatever causes certbot to complain about can be resolved? Like, if it's to make it not complain about not running as root... isn't there something with the python stuff that lets user packages to be used?
Maybe I'm just dumb but I'm baffled why this is a thing...
@LexiconCode commented on GitHub (Sep 8, 2024):
Workaround add an Additional Environment Variable on Truenas 24.10-Beta.1:
S6_STAGE2_HOOK=sed -i $d /etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh@Dremor commented on GitHub (Sep 14, 2024):
That's indeed the current (sometime buggy) workaround, but that won't resolve the issue itself.
The issue was caused by going from a python user package to a system package (see commit linked in the issue), which, if I understand it well, can only be run as root unless you chown it back, which take ages.
The question is, should we really chown the whole
/opt/certbot, or is it possible to do it another way, like chowning only a subset (like only the plugin directory, and only if there is a plugin installed) to reduce the time required, or, ideally, finding a better solution that the commit introducing this was trying to address.@skirsch commented on GitHub (Sep 23, 2024):
It's only 20 minutes startup time with HDD in a truenas array. WIth the fix, the startup time is 5 seconds.
@dasunsrule32 commented on GitHub (Sep 29, 2024):
This happens on straight docker as well on zfs.
@NicolasGoeddel commented on GitHub (Nov 1, 2024):
Wow, this is annoying. I just installed it on my TrueNAS 24.10 and while I was searching for a solution and writing this text it is still setting ownershop.
I fixed it by setting UID and GID to
0because everything in/opt/certbotbelongs toroot:rootanyway.@anakine22 commented on GitHub (Nov 1, 2024):
I experienced the same issue, thank you for your solution!
@zibler commented on GitHub (Nov 17, 2024):
Thank you! Finally, it starts up.
@lucky13820 commented on GitHub (Jan 2, 2025):
This works perfectly, thanks！
@CajuCLC commented on GitHub (Jan 14, 2025):
I just want to say thank you for this.
I am running TrueNAS 24.10 and was having the same issue, taking 5+ minutes for nginx proxy manager to start. I didn't want to change the UID and GID to run as root, so went with your recommendation. It started instantly.
I use the same pool as the apps, so there is no issues with speed (also nvme).
@NicolasGoeddel commented on GitHub (Jan 14, 2025):
Realistically changing the UID and GID does not harm security in any way because the container is running in its own namespace anyway.
Also: In
/opt/certbotare a lot of libraries that come with the container image. Copying it and mounting it again will prevent these libraries from being updated.@Korkman commented on GitHub (Feb 23, 2025):
Can confirm this is annoying. A route the devs may take:
Now if there is no volume mounted on /opt, the entire copy process would repeat on every run. To optimize for people who run with UID 0, just symlink the true location to the moved directory. So they get instant startup again.
Once there is a volume mounted on /opt, it can persist and non-root users get a much faster startup (rsync still has to stat all files to check for changes, but reading is much faster than modifying permissions).
@Korkman commented on GitHub (Mar 1, 2025):
Like this:
Dockerfile:
persist-permissions.sh:
Strictly speaking it isn't even necessary to use rsync and pass a volume for /opt. cp /opt + find + chown finishes in seconds. But the optimum is to now pass an empty volume to /opt.
@Korkman commented on GitHub (Mar 1, 2025):
I can transform this into a pull request if welcome
@Tsaukpaetra commented on GitHub (Mar 26, 2025):
Can't hurt. Might as well put it up there. :)
@nonamethanks commented on GitHub (Jun 29, 2025):
For me it's not just the first boot, it's every boot. It takes half a hour for this container to restart because it needs to reapply permissions every single time. How is it possible???
@zibler commented on GitHub (Jul 2, 2025):
Hello there,
I just ran into this issue. I pulled the latest version of the container and then everything stopped working. The container is stuck now in "setting ownership". I am not quite sure why it worked before?
I have been trying to get this working. I have waited 30 minutes to see if the container will finally boot up and nothing. I have tried setting the UID and GID to 0 and nothing either. I also tried setting the S6_STAGE2_HOOK environment variable and nothing. I know I can technically mount a volume and copy the contents of opt/certbot, but I don't really want to do that to avoid update issues in the future.
Any pointers or hints you could provide? I would greatly appreciate it.
@zibler commented on GitHub (Jul 2, 2025):
Forgot to mention. I am running this on ZFS on an SSD drive.
@zibler commented on GitHub (Jul 2, 2025):
Just noticed something. I experienced the issue with version 2.12.4. Decided to go back to version 2.12.3 and the container booted up right away. Don't know why, but I am sticking with this version for now
@phyco1991 commented on GitHub (Jul 2, 2025):
Yep definitely an issue in 2.12.4
I was getting "/etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh: line 41: syntax error: unexpected end of file" in the npm logs
After removing the S6_STAGE2_HOOK environment variable in Truenas I got stuck on "setting ownership" but as soon as I rolled back to 2.12.3 all worked immediately.
@daiv92 commented on GitHub (Jul 3, 2025):
Also updated to 2.12.4 and got stuck setting ownership - reverted to 2.12.3 and it's fine
@gquittet commented on GitHub (Jul 3, 2025):
Same issue on my side with 2.12.4 and ownership.
Rollback to 2.12.3 fixed the issue.
@focustense commented on GitHub (Jul 6, 2025):
This issue doesn't appear to be closed (yet?), but in searching for why the sed hook stopped working in 2.12.4, I saw that the
30-ownershipscript has been set to add a marker file that skips all subsequent runs after the first one (unless certbot data is updated, I guess, which makes sense). So the expected behavior after removing theS6_STAGE2_HOOKwould be for one slow start followed by fast starts afterward.Thought I'd mention this for anyone else stumbling onto the issue. I'm currently rolled back to 2.12.3 because there are some other bug reports and I'd rather wait 'til everyone agrees it's stabilized before updating again, but afterward it should be OK to remove the hook, and just be patient on the first startup.
@Mr-Newlove commented on GitHub (Jul 6, 2025):
updated to 2.12.4 and this started happening again to me, tried to roll back to 2.12.3 and it is still stuck for me.
both versions now:
Even after rolling back snapshots etc... stuck here, at least I assume this is the same issue
@NicolasGoeddel commented on GitHub (Jul 6, 2025):
Oh no. It is doing it again. What a bullshit. I guess I must remove the whole app and replace it with something better, e.g. traefik or something like that.
@Nightreaver commented on GitHub (Jul 6, 2025):
Do it! Or just do something yourself, then you can do it better....
I didnt have issues since I just mounted /opt month ago
@NicolasGoeddel commented on GitHub (Jul 6, 2025):
Which is not a proper solution. This way everything in /opt will not be updated.
I also fixed the issue months ago by changing UID and GID. It makes no sense that it starts
chown-ing it again now. It's just wrong to do that in the starting process. It must be fixed.@focustense commented on GitHub (Jul 6, 2025):
Let's be fair here, it is not NPM's fault that
opt/certbothas almost 8000 small files. That's how it arrives out of the gate when installing certbot on debian. Nor is it NPM's fault that whenever the container is rebuilt,/opt/certbotreverts to default ownership and has to bechowned again to work for non-root users; that's just how Docker works, all Docker builds with proper non-root support need some logic to fix ownership, they just usually don't have so many files to update.I think devs did the right thing in 2.12.4 to (a) only process directories that don't already have the correct ownership and (b) skip the ownership step after the first successful pass. So just let it run for a few minutes the first time, don't be impatient.
Though if I may make a suggestion, a common trick for
chown/chmod-ing a large number of files/directories is to do it in parallel, using GNUparallelorxargs -p. There might be a few degenerate cases where that doesn't help or even makes things a tad slower (i.e. single-core, single slow mechanical disk) but most people are probably running this in an environment with core capacity and spindles to spare (or on an SSD).Another possibility to ease the pain would be allowing us to disable certbot/letsencrypt entirely, either as an environment variable or as a trimmed-down docker build. Some of us don't use it, I personally have a third-party wildcard cert.
@edlman commented on GitHub (Jul 8, 2025):
I'm not sure from since the startup script /etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh of HA changed, so the original S6_STAGE2_HOOK doesn't work anymore. Original hook (I used) removed the last line of the script which was
chown -R $UID:$GID /opt/certbot, but now it's end of conditionfi. Removing this line will break up the script and you end up with "/etc/s6-overlay/s6-rc.d/prepare/30-ownership.sh: line 41: syntax error: unexpected end of file".I removed the hook and NPM started for almost 2 hours (on SSD disks!!!!!!!)
So I checked the changes in 30-ownership.sh script to find out it's checking existence of
/opt/certbot/.ownership_initialized. If this file exists (created later in the sctript), the ownership is not modified.So the solution is to use S6_STAGE2_HOOK with value
touch /opt/certbot/.ownership_initialized. It will skip all thechowncalls.@Leonid-Perkin commented on GitHub (Jul 9, 2025):
It really works!
Deployment is almost instant, even on very slow hardware.
@Dankitymao commented on GitHub (Jul 10, 2025):
As of June 9, 2025, an update to 30-ownership.sh now checks for the presence of an environment variable "SKIP_CERTBOT_OWNERSHIP"; if true, it skips the ownership settings for Certbot files, which is the part causing the delay. Tested with this environment variable set and it starts immediately again.