[GH-ISSUE #68] DNS Resolution is very slow for container names -- >10 seconds #28

Closed
opened 2026-02-26 04:33:49 +03:00 by kerem · 10 comments
Owner

Originally created by @ak2766 on GitHub (May 12, 2018).
Original GitHub issue: https://github.com/mageddo/dns-proxy-server/issues/68

I'm testing out your DNS Proxy Server but it appears to be very slow for docker container hostname resolution. I can resolve Internet hostnames in an instant but container hostname resolution takes over 10 seconds before I start getting replies to my ping requests.

I'm on Ubuntu 16.04.4 LTS if that matters.

Let me know what I can provide to help track down the issue.

Originally created by @ak2766 on GitHub (May 12, 2018). Original GitHub issue: https://github.com/mageddo/dns-proxy-server/issues/68 I'm testing out your DNS Proxy Server but it appears to be very slow for docker container hostname resolution. I can resolve Internet hostnames in an instant but container hostname resolution takes over 10 seconds before I start getting replies to my ping requests. I'm on Ubuntu 16.04.4 LTS if that matters. Let me know what I can provide to help track down the issue.
kerem 2026-02-26 04:33:49 +03:00
  • closed this issue
  • added the
    bug
    label
Author
Owner

@mageddo commented on GitHub (May 12, 2018):

@ak2766 Thanks for use DPS

You take 10 seconds in which scenario? by accessing the browser, by running a ping command, nslookup?

I never got any delay in DNS resolution from the browser or nslookup from terminal, but I notice some delay in ping command testing it right now, please confirm your test source, then we can investigate that.

Regards

<!-- gh-comment-id:388561999 --> @mageddo commented on GitHub (May 12, 2018): @ak2766 Thanks for use DPS You take 10 seconds in which scenario? by accessing the browser, by running a `ping` command, `nslookup`? I never got any delay in DNS resolution from the browser or nslookup from terminal, but I notice some delay in ping command testing it right now, please confirm your test source, then we can investigate that. Regards
Author
Owner

@ak2766 commented on GitHub (May 13, 2018):

Hi Mageddo.

Yes, it happens when I'm sending pings (I did mention in the post above, maybe you missed it).

I've started the dns-proxy-server as follows:

docker run -h dns.mageddo --name dns-proxy-server -p 5380:5380 -v /var/run/docker.sock:/var/run/docker.sock -v /etc/resolv.conf:/etc/resolv.conf -d defreitas/dns-proxy-server

I then started a CentOS image with the hostname pingclient.example.com as follows:
docker run -it --name pingclient -h pingclient.example.com -d centos

I then created an image based on CentOS with bind-utils and net-tools:

cat Dockerfile 
from centos
run yum install -y bind-utils net-tools
cmd ["/bin/bash"]

Then started it:
docker run -it --name pinger -h pinger.example.com -d pinger

I then pinged pingclient from the pinger with the following results:

docker exec -i pinger ping -c4 pingclient.example.com
PING pingclient.example.com (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=1 ttl=64 time=0.043 ms
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=2 ttl=64 time=0.062 ms
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=3 ttl=64 time=0.096 ms
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=4 ttl=64 time=0.102 ms

--- pingclient.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 12044ms
rtt min/avg/max/mdev = 0.043/0.075/0.102/0.026 ms

As you can see, it took 12.044s to send 4 ping most of which was spent waiting for DNS to resolve the name.

If I ping an internet location - www.google.com.au - I get responses very quickly - only 3 seconds as indicated below:

docker exec -i pinger ping -c4 www.google.com.au
PING www.google.com.au (172.217.25.163) 56(84) bytes of data.
64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=1 ttl=51 time=20.4 ms
64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=2 ttl=51 time=20.8 ms
64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=3 ttl=51 time=20.9 ms
64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=4 ttl=51 time=21.6 ms

--- www.google.com.au ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 20.408/20.970/21.686/0.472 ms

So, Definitely something odd going on with container name resolution.

<!-- gh-comment-id:388595572 --> @ak2766 commented on GitHub (May 13, 2018): Hi Mageddo. Yes, it happens when I'm sending pings (I did mention in the post above, maybe you missed it). I've started the `dns-proxy-server` as follows: ``` docker run -h dns.mageddo --name dns-proxy-server -p 5380:5380 -v /var/run/docker.sock:/var/run/docker.sock -v /etc/resolv.conf:/etc/resolv.conf -d defreitas/dns-proxy-server ``` I then started a `CentOS` image with the hostname `pingclient.example.com` as follows: ```docker run -it --name pingclient -h pingclient.example.com -d centos``` I then created an image based on `CentOS` with `bind-utils` and `net-tools`: ``` cat Dockerfile from centos run yum install -y bind-utils net-tools cmd ["/bin/bash"] ``` Then started it: ```docker run -it --name pinger -h pinger.example.com -d pinger``` I then pinged `pingclient` from the `pinger` with the following results: ``` docker exec -i pinger ping -c4 pingclient.example.com PING pingclient.example.com (172.17.0.3) 56(84) bytes of data. 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=1 ttl=64 time=0.043 ms 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=2 ttl=64 time=0.062 ms 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=3 ttl=64 time=0.096 ms 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=4 ttl=64 time=0.102 ms --- pingclient.example.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 12044ms rtt min/avg/max/mdev = 0.043/0.075/0.102/0.026 ms ``` As you can see, it took 12.044s to send 4 ping most of which was spent waiting for DNS to resolve the name. If I ping an internet location - `www.google.com.au` - I get responses very quickly - only 3 seconds as indicated below: ``` docker exec -i pinger ping -c4 www.google.com.au PING www.google.com.au (172.217.25.163) 56(84) bytes of data. 64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=1 ttl=51 time=20.4 ms 64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=2 ttl=51 time=20.8 ms 64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=3 ttl=51 time=20.9 ms 64 bytes from syd09s13-in-f163.1e100.net (172.217.25.163): icmp_seq=4 ttl=51 time=21.6 ms --- www.google.com.au ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 20.408/20.970/21.686/0.472 ms ``` So, Definitely something odd going on with container name resolution.
Author
Owner

@ak2766 commented on GitHub (May 13, 2018):

Interestingly, dig returns results very quickly:

$ docker exec -i pinger dig pingclient.example.com

; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> pingclient.example.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12692
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;pingclient.example.com.		IN	A

;; ANSWER SECTION:
pingclient.example.com.	0	IN	A	172.17.0.3

;; Query time: 0 msec
;; SERVER: 172.17.0.2#53(172.17.0.2)
;; WHEN: Sun May 13 02:08:12 UTC 2018
;; MSG SIZE  rcvd: 78

Yet, pinging immediately after still results in a long wait before ping replies start coming back:

$ docker exec -i pinger ping -c4 pingclient.example.com
PING pingclient.example.com (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=2 ttl=64 time=0.042 ms
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=3 ttl=64 time=0.052 ms
64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=4 ttl=64 time=0.082 ms

--- pingclient.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 12043ms
rtt min/avg/max/mdev = 0.042/0.055/0.082/0.016 ms
<!-- gh-comment-id:388595903 --> @ak2766 commented on GitHub (May 13, 2018): Interestingly, `dig` returns results very quickly: ``` $ docker exec -i pinger dig pingclient.example.com ; <<>> DiG 9.9.4-RedHat-9.9.4-61.el7 <<>> pingclient.example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12692 ;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;pingclient.example.com. IN A ;; ANSWER SECTION: pingclient.example.com. 0 IN A 172.17.0.3 ;; Query time: 0 msec ;; SERVER: 172.17.0.2#53(172.17.0.2) ;; WHEN: Sun May 13 02:08:12 UTC 2018 ;; MSG SIZE rcvd: 78 ``` Yet, pinging immediately after still results in a long wait before ping replies start coming back: ``` $ docker exec -i pinger ping -c4 pingclient.example.com PING pingclient.example.com (172.17.0.3) 56(84) bytes of data. 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=1 ttl=64 time=0.044 ms 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=2 ttl=64 time=0.042 ms 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=3 ttl=64 time=0.052 ms 64 bytes from 172.17.0.3 (172.17.0.3): icmp_seq=4 ttl=64 time=0.082 ms --- pingclient.example.com ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 12043ms rtt min/avg/max/mdev = 0.042/0.055/0.082/0.016 ms ```
Author
Owner

@ak2766 commented on GitHub (May 13, 2018):

For completeness sake, when I ping the IP Address, response is immediate:

docker exec -i pinger ping -c4 172.17.0.3
PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data.
64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.064 ms
64 bytes from 172.17.0.3: icmp_seq=3 ttl=64 time=0.127 ms
64 bytes from 172.17.0.3: icmp_seq=4 ttl=64 time=0.109 ms

--- 172.17.0.3 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3065ms
rtt min/avg/max/mdev = 0.058/0.089/0.127/0.030 ms
<!-- gh-comment-id:388597491 --> @ak2766 commented on GitHub (May 13, 2018): For completeness sake, when I ping the IP Address, response is immediate: ``` docker exec -i pinger ping -c4 172.17.0.3 PING 172.17.0.3 (172.17.0.3) 56(84) bytes of data. 64 bytes from 172.17.0.3: icmp_seq=1 ttl=64 time=0.058 ms 64 bytes from 172.17.0.3: icmp_seq=2 ttl=64 time=0.064 ms 64 bytes from 172.17.0.3: icmp_seq=3 ttl=64 time=0.127 ms 64 bytes from 172.17.0.3: icmp_seq=4 ttl=64 time=0.109 ms --- 172.17.0.3 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3065ms rtt min/avg/max/mdev = 0.058/0.089/0.127/0.030 ms ```
Author
Owner

@ak2766 commented on GitHub (May 13, 2018):

FYI: I've just repeated the experiment with a user defined network and ping is now super responsive:

I created a user defined network as below:
docker network create dockernet

I then added --network dockernet option as I started the containers above.

This site was instrumental in my experimentation:
User Defined Bridges

Cheers,
ak.

<!-- gh-comment-id:388626620 --> @ak2766 commented on GitHub (May 13, 2018): FYI: I've just repeated the experiment with a user defined network and ping is now super responsive: I created a user defined network as below: `docker network create dockernet` I then added `--network dockernet` option as I started the containers above. This site was instrumental in my experimentation: [User Defined Bridges](https://docs.docker.com/network/bridge/#differences-between-user-defined-bridges-and-the-default-bridge) Cheers, ak.
Author
Owner

@ak2766 commented on GitHub (May 15, 2018):

Update:

It appears that the fast response is not always guaranteed. Sometimes super fast other times I get that same ~10 second delay before I get my responses.

<!-- gh-comment-id:389328617 --> @ak2766 commented on GitHub (May 15, 2018): Update: It appears that the fast response is not always guaranteed. Sometimes super fast other times I get that same ~10 second delay before I get my responses.
Author
Owner

@mageddo commented on GitHub (May 16, 2018):

@ak2766 I think I got it.

Basically when you ping a hostname it will do a second question to DNS server asking for reverse DNS lookup eg. 2.0.18.172.in-addr.arpa., as it doesn't exists, remote server returns an error code, DPS code was with a bug that returns nothing when it receives an error code from remote server then ping timeouts after 10s and keep going.

I will pre-release version 2.5.1 and keep testing.

Thank you very much for your contribution.

<!-- gh-comment-id:389393426 --> @mageddo commented on GitHub (May 16, 2018): @ak2766 I think I got it. Basically when you ping a hostname it will do a second question to DNS server asking for reverse DNS lookup eg. `2.0.18.172.in-addr.arpa.`, as it doesn't exists, remote server returns an error code, DPS code was with a bug that returns nothing when it receives an error code from remote server then ping timeouts after 10s and keep going. I will pre-release version **2.5.1** and keep testing. Thank you very much for your contribution.
Author
Owner

@mageddo commented on GitHub (May 16, 2018):

PR

<!-- gh-comment-id:389394311 --> @mageddo commented on GitHub (May 16, 2018): [PR](https://github.com/mageddo/dns-proxy-server/pull/69)
Author
Owner

@ak2766 commented on GitHub (May 19, 2018):

Thank you very much for your contribution.

Thank you much more for creating a tool that makes my life really easy while getting to know Docker.

<!-- gh-comment-id:390366556 --> @ak2766 commented on GitHub (May 19, 2018): > Thank you very much for your contribution. Thank you much more for creating a tool that makes my life really easy while getting to know Docker.
Author
Owner

@ak2766 commented on GitHub (May 19, 2018):

I've been running this peridically (stopping and starting all containers prior to testing):

for u in {1..1000}; do ping -f -c 100 pingclient.example.com 2>&1 | egrep transmitted; done

I haven't seen a number > 2ms.

Solid fix muchly appreciated.

<!-- gh-comment-id:390368693 --> @ak2766 commented on GitHub (May 19, 2018): I've been running this peridically (stopping and starting all containers prior to testing): `for u in {1..1000}; do ping -f -c 100 pingclient.example.com 2>&1 | egrep transmitted; done` I haven't seen a number > 2ms. Solid fix muchly appreciated.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/dns-proxy-server-mageddo#28
No description provided.