[GH-ISSUE #182] acme-dns not appearing to be running as service #77

Closed
opened 2026-03-13 15:41:34 +03:00 by kerem · 6 comments
Owner

Originally created by @icelava on GitHub (Sep 10, 2019).
Original GitHub issue: https://github.com/acme-dns/acme-dns/issues/182

When I first tested as a user instance in terminal foreground, /root/go/bin/acme-dns was able to run as expected. Now having followed the re-configuring instructions to run acme-dns as a systemd service with a less-privileged user, I suspect there's something preventing it from starting up properly; there's no evidence of a process running and listening to the exact ports it should be operating on. nestat -tlpn does not show any acme-dns process

If I manually launch /usr/local/bin/acme-dns/acme-dns however, only then would the listened ports show up to the correct process.

# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 192.168.0.21:53         0.0.0.0:*               LISTEN      6067/./acme-dns
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      883/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1329/sshd
tcp6       0      0 :::80                   :::*                    LISTEN      6067/./acme-dns
tcp6       0      0 :::22                   :::*                    LISTEN      1329/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      6067/./acme-dns
udp        0      0 192.168.0.21:53         0.0.0.0:*                           6067/./acme-dns
udp        0      0 127.0.0.53:53           0.0.0.0:*                           883/systemd-resolve

Although I'm also puzzled why IPv4 port 80/443 not bound? Even though I can reach them...

Originally created by @icelava on GitHub (Sep 10, 2019). Original GitHub issue: https://github.com/acme-dns/acme-dns/issues/182 When I first tested as a user instance in terminal foreground, /root/go/bin/acme-dns was able to run as expected. Now having followed the re-configuring instructions to run acme-dns as a systemd service with a less-privileged user, I suspect there's something preventing it from starting up properly; there's no evidence of a process running and listening to the exact ports it should be operating on. nestat -tlpn does not show any acme-dns process If I manually launch /usr/local/bin/acme-dns/acme-dns however, only then would the listened ports show up to the correct process. ``` # netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 192.168.0.21:53 0.0.0.0:* LISTEN 6067/./acme-dns tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 883/systemd-resolve tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1329/sshd tcp6 0 0 :::80 :::* LISTEN 6067/./acme-dns tcp6 0 0 :::22 :::* LISTEN 1329/sshd tcp6 0 0 :::443 :::* LISTEN 6067/./acme-dns udp 0 0 192.168.0.21:53 0.0.0.0:* 6067/./acme-dns udp 0 0 127.0.0.53:53 0.0.0.0:* 883/systemd-resolve ``` Although I'm also puzzled why IPv4 port 80/443 not bound? Even though I can reach them...
kerem closed this issue 2026-03-13 15:41:39 +03:00
Author
Owner

@Mikaela commented on GitHub (Sep 10, 2019):

Although I'm also puzzled why IPv4 port 80/443 not bound? Even though I can reach them...

I guess you have net.ipv6.bindv6only = 0 (with sysctl/kernel) which makes binding :: also bind 0.0.0.0 unless the socket especially binds to :: from what I have understood.

I personally set it to 1 to disable IPv4 mapped IPv6.

As you see this I wonder if on my system acme-dns would be IPv6-only, I haven't yet ran it personally, but am considering starting to use it.

<!-- gh-comment-id:530104034 --> @Mikaela commented on GitHub (Sep 10, 2019): > Although I'm also puzzled why IPv4 port 80/443 not bound? Even though I can reach them... I guess you have `net.ipv6.bindv6only = 0` (with sysctl/kernel) which makes binding :: also bind 0.0.0.0 unless the socket especially binds to :: from what I have understood. I personally set it to 1 to disable [IPv4 mapped IPv6](https://serverfault.com/q/408667). As you see this I wonder if on my system acme-dns would be IPv6-only, I haven't yet ran it personally, but am considering starting to use it.
Author
Owner

@icelava commented on GitHub (Sep 11, 2019):

Ok I can check on that later, but the more critical issue is acme-dns.service does not appear to run in the background despite systemctl start acme-dns.service ran without any shown error.

<!-- gh-comment-id:530196741 --> @icelava commented on GitHub (Sep 11, 2019): Ok I can check on that later, but the more critical issue is acme-dns.service does not appear to run in the background despite `systemctl start acme-dns.service` ran without any shown error.
Author
Owner

@icelava commented on GitHub (Sep 11, 2019):

I guess you have net.ipv6.bindv6only = 0 (with sysctl/kernel) which makes binding :: also bind 0.0.0.0 unless the socket especially binds to :: from what I have understood.

That directive is supposed to be in /etc/sysctl.conf, yes? I don't see such in the file and practically everything is commented out.

cat sysctl.conf | grep ipv6
#net.ipv6.conf.all.forwarding=1
#net.ipv6.conf.all.accept_redirects = 0
#net.ipv6.conf.all.accept_source_route = 0

When I start up Nginx (which) has to be shut down to avoid conflicting port bindings), it appears to be able to bind ports 80/443 on IPv4 and IPv6 just fine.

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6412/nginx: master
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      883/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1329/sshd
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      6412/nginx: master
tcp6       0      0 :::80                   :::*                    LISTEN      6412/nginx: master
tcp6       0      0 :::22                   :::*                    LISTEN      1329/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      6412/nginx: master
udp        0      0 127.0.0.53:53           0.0.0.0:*                           883/systemd-resolve

I believe prior to this re-configuration to run ACME-DNS as non-root it was able to bind IPv4 0.0.0.0 similarly.

<!-- gh-comment-id:530212136 --> @icelava commented on GitHub (Sep 11, 2019): > I guess you have `net.ipv6.bindv6only = 0` (with sysctl/kernel) which makes binding :: also bind 0.0.0.0 unless the socket especially binds to :: from what I have understood. That directive is supposed to be in /etc/sysctl.conf, yes? I don't see such in the file and practically everything is commented out. ``` cat sysctl.conf | grep ipv6 #net.ipv6.conf.all.forwarding=1 #net.ipv6.conf.all.accept_redirects = 0 #net.ipv6.conf.all.accept_source_route = 0 ``` When I start up Nginx (which) has to be shut down to avoid conflicting port bindings), it appears to be able to bind ports 80/443 on IPv4 and IPv6 just fine. ``` Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6412/nginx: master tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 883/systemd-resolve tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1329/sshd tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 6412/nginx: master tcp6 0 0 :::80 :::* LISTEN 6412/nginx: master tcp6 0 0 :::22 :::* LISTEN 1329/sshd tcp6 0 0 :::443 :::* LISTEN 6412/nginx: master udp 0 0 127.0.0.53:53 0.0.0.0:* 883/systemd-resolve ``` I believe prior to this re-configuration to run ACME-DNS as non-root it was able to bind IPv4 0.0.0.0 similarly.
Author
Owner

@icelava commented on GitHub (Sep 11, 2019):

Ok i just learnt how to peek into systemd service startup logs, with journalctl --since "10 minutes ago"

Sep 11 04:27:52 ngnix systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.
Sep 11 04:27:52 ngnix acme-dns[6657]: time="2019-09-11T04:27:52Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg
Sep 11 04:27:52 ngnix acme-dns[6657]: time="2019-09-11T04:27:52Z" level=error msg="Could not open database [unable to open database file]"
Sep 11 04:27:52 ngnix systemd[1]: acme-dns.service: Main process exited, code=exited, status=1/FAILURE
Sep 11 04:27:52 ngnix systemd[1]: acme-dns.service: Failed with result 'exit-code'.

Hmmm, but I did grant acme-dns user and group rw permission to /var/lib/acme-dns/acme-dns.db so what could be the problem? Manually running as root is fine.

<!-- gh-comment-id:530215826 --> @icelava commented on GitHub (Sep 11, 2019): Ok i just learnt how to peek into systemd service startup logs, with journalctl --since "10 minutes ago" ``` Sep 11 04:27:52 ngnix systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely. Sep 11 04:27:52 ngnix acme-dns[6657]: time="2019-09-11T04:27:52Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg Sep 11 04:27:52 ngnix acme-dns[6657]: time="2019-09-11T04:27:52Z" level=error msg="Could not open database [unable to open database file]" Sep 11 04:27:52 ngnix systemd[1]: acme-dns.service: Main process exited, code=exited, status=1/FAILURE Sep 11 04:27:52 ngnix systemd[1]: acme-dns.service: Failed with result 'exit-code'. ``` Hmmm, but I did grant acme-dns user and group rw permission to /var/lib/acme-dns/acme-dns.db so what could be the problem? Manually running as root is fine.
Author
Owner

@Mikaela commented on GitHub (Sep 11, 2019):

That directive is supposed to be in /etc/sysctl.conf, yes? I don't see such in the file and practically everything is commented out.

Yes or alternatively /etc/sysctl.d/<whatever>.conf, it defaults to 0 if not specified. If you want to confirm, try running sysctl net.ipv6.bindv6only

<!-- gh-comment-id:530268921 --> @Mikaela commented on GitHub (Sep 11, 2019): > That directive is supposed to be in /etc/sysctl.conf, yes? I don't see such in the file and practically everything is commented out. Yes or alternatively `/etc/sysctl.d/<whatever>.conf`, it defaults to 0 if not specified. If you want to confirm, try running `sysctl net.ipv6.bindv6only`
Author
Owner

@icelava commented on GitHub (Sep 11, 2019):

Just performed the ACME-DNS systemd service re-config on another server, and it appears to launch and remain running just fine. I noticed the slight difference in permission set for /var/lib/acme-dns directory itself

Working: drwxr-xr-x 2 acme-dns acme-dns 4096 Sep 11 06:39 acme-dns
Faulty: drwxr-x--- 2 acme-dns acme-dns 4096 Sep 10 18:44 acme-dns

Does "others" need to have rx to the directory? Anyway I added that in but still failure to open database file.

Sep 11 07:47:13 ngnix acme-dns[7621]: time="2019-09-11T07:47:13Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg
Sep 11 07:47:13 ngnix acme-dns[7621]: time="2019-09-11T07:47:13Z" level=error msg="Could not open database [unable to open database file]"

BUT, on inspecting stuff slowly one at a time, I found /var/lib permissions fishy

root@ngnix:/var# ls -ald lib
drwxr----- 39 root root 4096 Sep  2 17:56 lib
root@ngnix:/var# chmod go+rx lib
root@ngnix:/var# ls -ald lib
drwxr-xr-x 39 root root 4096 Sep  2 17:56 lib

I don't remember attempting to control permissions at the /var/lib level though (why should I?). Anyway that appears to be the culprit!

Sep 11 08:11:46 ngnix systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Connected to database"
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=a.domain.net recordtype=A
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=b.domain.net recordtype=A
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=a.domain.net recordtype=NS
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=a.domain.net recordtype=SOA
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Opening HTTP port for autocert" autocert" autocerthost=":80" domain=a.domain.net
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Listening HTTPS, using certificate from autocert" domain=a.domain.net host=":443"
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Listening DNS" addr="192.168.0.21:53" proto=udp
Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Listening DNS" addr="192.168.0.21:53" proto=tcp
<!-- gh-comment-id:530277494 --> @icelava commented on GitHub (Sep 11, 2019): Just performed the ACME-DNS systemd service re-config on another server, and it appears to launch and remain running just fine. I noticed the slight difference in permission set for /var/lib/acme-dns directory itself Working: **drwxr-xr-x** 2 acme-dns acme-dns 4096 Sep 11 06:39 acme-dns Faulty: **drwxr-x---** 2 acme-dns acme-dns 4096 Sep 10 18:44 acme-dns Does "others" need to have rx to the directory? Anyway I added that in but still failure to open database file. ``` Sep 11 07:47:13 ngnix acme-dns[7621]: time="2019-09-11T07:47:13Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg Sep 11 07:47:13 ngnix acme-dns[7621]: time="2019-09-11T07:47:13Z" level=error msg="Could not open database [unable to open database file]" ``` BUT, on inspecting stuff slowly one at a time, I found /var/lib permissions fishy ``` root@ngnix:/var# ls -ald lib drwxr----- 39 root root 4096 Sep 2 17:56 lib root@ngnix:/var# chmod go+rx lib root@ngnix:/var# ls -ald lib drwxr-xr-x 39 root root 4096 Sep 2 17:56 lib ``` I don't remember attempting to control permissions at the **/var/lib** level though (why should I?). Anyway that appears to be the culprit! ``` Sep 11 08:11:46 ngnix systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely. Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Connected to database" Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=a.domain.net recordtype=A Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=b.domain.net recordtype=A Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=a.domain.net recordtype=NS Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Adding new record to domain" domain=a.domain.net recordtype=SOA Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=debug msg="Opening HTTP port for autocert" autocert" autocerthost=":80" domain=a.domain.net Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Listening HTTPS, using certificate from autocert" domain=a.domain.net host=":443" Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Listening DNS" addr="192.168.0.21:53" proto=udp Sep 11 08:11:46 ngnix acme-dns[7652]: time="2019-09-11T08:11:46Z" level=info msg="Listening DNS" addr="192.168.0.21:53" proto=tcp ```
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/acme-dns#77
No description provided.