[GH-ISSUE #191] connection timed out issue #78

New issue

Closed

opened 2026-02-26 04:33:58 +03:00 by kerem · 1 comment

kerem commented

2026-02-26 04:33:58 +03:00

Owner

Originally created by @feiyanke on GitHub (Feb 13, 2020).
Original GitHub issue: https://github.com/mageddo/dns-proxy-server/issues/191

What is Happening

I use DPS with KubeEdge (an edge computing platform with k8s)
When start DPS with running kubeedge PODs(with hostname: nginx-edge), nslookup works fine, however, when PODs restart,
nslookup get connection timed out error.

[root@edge-b305dc1eee21 ~]# nslookup nginx-edge
;; connection timed out; no servers could be reached

Specs:

OS: CenOS 7
Docker Version: 19.03.5
DPS Version: 2.18.7

Logs:

2020/02/12 21:41:44.868198 INFO f=docker.go:104 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents uuid=841367b933 status=resolved-hosts, action=die, hostnames=[nginx-edge]
2020/02/12 21:41:44.870127 INFO f=docker.go:104 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents uuid=efaf7be9d9 status=resolved-hosts, action=stop, hostnames=[nginx-edge]
2020/02/12 21:42:26.043801 INFO f=docker.go:104 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents uuid=90bbeb0071 status=resolved-hosts, action=start, hostnames=[nginx-edge]
2020/02/12 21:42:42.276827 ERROR f=panic.go:523 pkg=runtime m=gopanic uuid=534040c913 status=fatal-error-handling-question, req=[{Name:nginx-edge. Qtype:1 Qclass:1}], err=runtime error: index out of range, stack=goroutine 193 [running]:
runtime/debug.Stack(0xc0003437a0, 0x7f3260, 0xbbd940)
        /usr/local/go/src/runtime/debug/stack.go:24 +0x9d
main.handleQuestion.func1(0x8fedc0, 0xc000171680, 0xc0000b47e0)
        /app/src/github.com/mageddo/dns-proxy-server/dns.go:32 +0x5a
panic(0x7f3260, 0xbbd940)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
github.com/mageddo/dns-proxy-server/proxy.DockerDnsSolver.getMsg(0x902920, 0xc00009c4e0, 0xc000099311, 0xa, 0xc000099140, 0xb, 0xc000010001, 0xc000343998)
        /app/src/github.com/mageddo/dns-proxy-server/proxy/docker.go:47 +0x3d9
github.com/mageddo/dns-proxy-server/proxy.DockerDnsSolver.doSolve(0x902920, 0xc00009c4e0, 0x8fedc0, 0xc000171680, 0xc000099311, 0xa, 0xc000099140, 0xb, 0xc000010001, 0xc000343a58, ...)
        /app/src/github.com/mageddo/dns-proxy-server/proxy/docker.go:36 +0x205
github.com/mageddo/dns-proxy-server/proxy.DockerDnsSolver.Solve(0x902920, 0xc00009c4e0, 0x8fedc0, 0xc000171680, 0xc000099140, 0xb, 0x10001, 0x0, 0x8f6bc0, 0xc000021820)
        /app/src/github.com/mageddo/dns-proxy-server/proxy/docker.go:24 +0x287
github.com/mageddo/dns-proxy-server/proxy.(*DefaultDnsSolverFactory).Solve(0xbe7398, 0x8fedc0, 0xc000171680, 0xc000099140, 0xb, 0xc000010001, 0xc0002756c0, 0x4, 0x4, 0x50, ...)
        /app/src/github.com/mageddo/dns-proxy-server/proxy/factory.go:19 +0xd8
github.com/mageddo/dns-proxy-server/proxy.(*CnameDnsSolverFactory).Solve(0xc000343db0, 0x8fedc0, 0xc000171680, 0xc000099140, 0xb, 0x10001, 0xc0002756c0, 0x4, 0x4, 0x715b25, ...)
        /app/src/github.com/mageddo/dns-proxy-server/proxy/factory.go:48 +0xbd
main.handleQuestion(0x902a40, 0xc000356b00, 0xc0000b47e0)
        /app/src/github.com/mageddo/dns-proxy-server/dns.go:51 +0x48e
github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.HandlerFunc.ServeDNS(0x88a430, 0x902a40, 0xc000356b00, 0xc0000b47e0)
        /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:84 +0x44
github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.(*ServeMux).ServeDNS(0xc0002fc0e0, 0x902a40, 0xc000356b00, 0xc0000b47e0)
        /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:210 +0x63
github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.(*Server).serve(0xc000306750, 0x8face0, 0xc000171650, 0x8f6c20, 0xc0002fc0e0, 0xc0003f8600, 0x1c, 0x200, 0xc0000100a0, 0xc00008e460, ...)
        /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:579 +0x2a4
created by github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.(*Server).serveUDP
        /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:533 +0x29e

Docker Env

[root@edge-b305dc1eee21 ~]# docker ps
CONTAINER ID        IMAGE                COMMAND                  CREATED              STATUS              PORTS               NAMES
62cc3aeef11b        5fad07aba15a         "nginx -g 'daemon of…"   About a minute ago   Up About a minute                       k8s_nginx_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_4
2ba359afa65d        kubeedge/pause:3.1   "/pause"                 12 minutes ago       Up 12 minutes                           k8s_POD_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_1

[root@edge-b305dc1eee21 ~]# docker inspect 62cc3aeef11b
[
    {
        "Id": "62cc3aeef11bbde0466ec03c2e1d3fe2e0b0380b732bea915a71954a263342fb",
        "Created": "2020-02-13T02:42:25.851751824Z",
        "Path": "nginx",
        "Args": [
            "-g",
            "daemon off;"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 9019,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-02-13T02:42:26.038034896Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:5fad07aba15a1047c94b9818de93093310ed1bf2db98c185f5f8486a6e53f434",
        "ResolvConfPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/hostname",
        "HostsPath": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/etc-hosts",
        "LogPath": "/var/lib/docker/containers/62cc3aeef11bbde0466ec03c2e1d3fe2e0b0380b732bea915a71954a263342fb/62cc3aeef11bbde0466ec03c2e1d3fe2e0b0380b732bea915a71954a263342fb-json.log",
        "Name": "/k8s_nginx_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_4",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/volumes/kubernetes.io~secret/default-token-dmx9w:/var/run/secrets/kubernetes.io/serviceaccount:ro,rslave",
                "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/etc-hosts:/etc/hosts",
                "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/containers/nginx/02a3148d:/dev/termination-log"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "container:2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c",
            "PortBindings": null,
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "container:2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 1000,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 2,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "cgroupfs.slice",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/asound",
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23-init/diff:/var/lib/docker/overlay2/4427ae02f52db721144b352ce214510061000c68e7786411e382212c9e619684/diff:/var/lib/docker/overlay2/73d0f6939e7114647ac4d448b901dc889ccb859af4ac027ae3223bd240ee5d3d/diff",
                "MergedDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23/merged",
                "UpperDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23/diff",
                "WorkDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/volumes/kubernetes.io~secret/default-token-dmx9w",
                "Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
                "Mode": "ro,rslave",
                "RW": false,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/containers/nginx/02a3148d",
                "Destination": "/dev/termination-log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "nginx-edge",
            "Domainname": "",
            "User": "0",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NGINX_VERSION=1.16.1",
                "NJS_VERSION=0.3.8",
                "PKG_RELEASE=1"
            ],
            "Cmd": [
                "nginx",
                "-g",
                "daemon off;"
            ],
            "Healthcheck": {
                "Test": [
                    "NONE"
                ]
            },
            "ArgsEscaped": true,
            "Image": "sha256:5fad07aba15a1047c94b9818de93093310ed1bf2db98c185f5f8486a6e53f434",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "annotation.io.kubernetes.container.hash": "121b46b1",
                "annotation.io.kubernetes.container.restartCount": "4",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/default_nginx-rr9z6_53ccb836-4959-41cd-ac29-19eddc8aa0c5/nginx/4.log",
                "io.kubernetes.container.name": "nginx",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "nginx-rr9z6",
                "io.kubernetes.pod.namespace": "default",
                "io.kubernetes.pod.uid": "53ccb836-4959-41cd-ac29-19eddc8aa0c5",
                "io.kubernetes.sandbox.id": "2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c",
                "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>"
            },
            "StopSignal": "SIGTERM"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {}
        }
    }
]

[root@edge-b305dc1eee21 ~]# docker inspect 2ba359afa65d
[
    {
        "Id": "2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c",
        "Created": "2020-02-13T02:31:10.641111343Z",
        "Path": "/pause",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 8490,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-02-13T02:31:10.943041797Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e",
        "ResolvConfPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/hostname",
        "HostsPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/hosts",
        "LogPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c-json.log",
        "Name": "/k8s_POD_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_1",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": -998,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": [
                "seccomp=unconfined"
            ],
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 2,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "cgroupfs.slice",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/asound",
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9-init/diff:/var/lib/docker/overlay2/07e6032dd42ab36194553def71e1c675ab62fc21cd1ec184a33e8d34efa879ba/diff",
                "MergedDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9/merged",
                "UpperDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9/diff",
                "WorkDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [],
        "Config": {
            "Hostname": "nginx-edge",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "kubeedge/pause:3.1",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": [
                "/pause"
            ],
            "OnBuild": null,
            "Labels": {
                "app": "nginx",
                "controller-revision-hash": "557488dfcb",
                "io.kubernetes.container.name": "POD",
                "io.kubernetes.docker.type": "podsandbox",
                "io.kubernetes.pod.name": "nginx-rr9z6",
                "io.kubernetes.pod.namespace": "default",
                "io.kubernetes.pod.uid": "53ccb836-4959-41cd-ac29-19eddc8aa0c5",
                "pod-template-generation": "1"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "aa07ae52799da38a3e36819f3704f74b11256d7b182269c4c43018491fcbb7e2",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/aa07ae52799d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "3662c7489c804530f8f97857f759de00b1a6555a48eeb455ebe745a27ab752ce",
            "Gateway": "172.17.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "02:42:ac:11:00:02",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "6ce2b7024fed591c6efde888dc4573e83774924f55c81035a89b6fefa9086c45",
                    "EndpointID": "3662c7489c804530f8f97857f759de00b1a6555a48eeb455ebe745a27ab752ce",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

Originally created by @feiyanke on GitHub (Feb 13, 2020). Original GitHub issue: https://github.com/mageddo/dns-proxy-server/issues/191 ### What is Happening * I use DPS with KubeEdge (an edge computing platform with k8s) * When start DPS with running kubeedge PODs(with hostname: nginx-edge), nslookup works fine, however, when PODs restart, nslookup get connection timed out error. ``` [root@edge-b305dc1eee21 ~]# nslookup nginx-edge ;; connection timed out; no servers could be reached ``` **Specs:** - OS: CenOS 7 - Docker Version: 19.03.5 - DPS Version: 2.18.7 **Logs:** ``` 2020/02/12 21:41:44.868198 INFO f=docker.go:104 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents uuid=841367b933 status=resolved-hosts, action=die, hostnames=[nginx-edge] 2020/02/12 21:41:44.870127 INFO f=docker.go:104 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents uuid=efaf7be9d9 status=resolved-hosts, action=stop, hostnames=[nginx-edge] 2020/02/12 21:42:26.043801 INFO f=docker.go:104 pkg=github.com/mageddo/dns-proxy-server/events/docker m=HandleDockerEvents uuid=90bbeb0071 status=resolved-hosts, action=start, hostnames=[nginx-edge] 2020/02/12 21:42:42.276827 ERROR f=panic.go:523 pkg=runtime m=gopanic uuid=534040c913 status=fatal-error-handling-question, req=[{Name:nginx-edge. Qtype:1 Qclass:1}], err=runtime error: index out of range, stack=goroutine 193 [running]: runtime/debug.Stack(0xc0003437a0, 0x7f3260, 0xbbd940) /usr/local/go/src/runtime/debug/stack.go:24 +0x9d main.handleQuestion.func1(0x8fedc0, 0xc000171680, 0xc0000b47e0) /app/src/github.com/mageddo/dns-proxy-server/dns.go:32 +0x5a panic(0x7f3260, 0xbbd940) /usr/local/go/src/runtime/panic.go:522 +0x1b5 github.com/mageddo/dns-proxy-server/proxy.DockerDnsSolver.getMsg(0x902920, 0xc00009c4e0, 0xc000099311, 0xa, 0xc000099140, 0xb, 0xc000010001, 0xc000343998) /app/src/github.com/mageddo/dns-proxy-server/proxy/docker.go:47 +0x3d9 github.com/mageddo/dns-proxy-server/proxy.DockerDnsSolver.doSolve(0x902920, 0xc00009c4e0, 0x8fedc0, 0xc000171680, 0xc000099311, 0xa, 0xc000099140, 0xb, 0xc000010001, 0xc000343a58, ...) /app/src/github.com/mageddo/dns-proxy-server/proxy/docker.go:36 +0x205 github.com/mageddo/dns-proxy-server/proxy.DockerDnsSolver.Solve(0x902920, 0xc00009c4e0, 0x8fedc0, 0xc000171680, 0xc000099140, 0xb, 0x10001, 0x0, 0x8f6bc0, 0xc000021820) /app/src/github.com/mageddo/dns-proxy-server/proxy/docker.go:24 +0x287 github.com/mageddo/dns-proxy-server/proxy.(*DefaultDnsSolverFactory).Solve(0xbe7398, 0x8fedc0, 0xc000171680, 0xc000099140, 0xb, 0xc000010001, 0xc0002756c0, 0x4, 0x4, 0x50, ...) /app/src/github.com/mageddo/dns-proxy-server/proxy/factory.go:19 +0xd8 github.com/mageddo/dns-proxy-server/proxy.(*CnameDnsSolverFactory).Solve(0xc000343db0, 0x8fedc0, 0xc000171680, 0xc000099140, 0xb, 0x10001, 0xc0002756c0, 0x4, 0x4, 0x715b25, ...) /app/src/github.com/mageddo/dns-proxy-server/proxy/factory.go:48 +0xbd main.handleQuestion(0x902a40, 0xc000356b00, 0xc0000b47e0) /app/src/github.com/mageddo/dns-proxy-server/dns.go:51 +0x48e github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.HandlerFunc.ServeDNS(0x88a430, 0x902a40, 0xc000356b00, 0xc0000b47e0) /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:84 +0x44 github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.(*ServeMux).ServeDNS(0xc0002fc0e0, 0x902a40, 0xc000356b00, 0xc0000b47e0) /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:210 +0x63 github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.(*Server).serve(0xc000306750, 0x8face0, 0xc000171650, 0x8f6c20, 0xc0002fc0e0, 0xc0003f8600, 0x1c, 0x200, 0xc0000100a0, 0xc00008e460, ...) /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:579 +0x2a4 created by github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns.(*Server).serveUDP /app/src/github.com/mageddo/dns-proxy-server/vendor/github.com/miekg/dns/server.go:533 +0x29e ``` **Docker Env** ``` [root@edge-b305dc1eee21 ~]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 62cc3aeef11b 5fad07aba15a "nginx -g 'daemon of…" About a minute ago Up About a minute k8s_nginx_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_4 2ba359afa65d kubeedge/pause:3.1 "/pause" 12 minutes ago Up 12 minutes k8s_POD_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_1 [root@edge-b305dc1eee21 ~]# docker inspect 62cc3aeef11b [ { "Id": "62cc3aeef11bbde0466ec03c2e1d3fe2e0b0380b732bea915a71954a263342fb", "Created": "2020-02-13T02:42:25.851751824Z", "Path": "nginx", "Args": [ "-g", "daemon off;" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 9019, "ExitCode": 0, "Error": "", "StartedAt": "2020-02-13T02:42:26.038034896Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:5fad07aba15a1047c94b9818de93093310ed1bf2db98c185f5f8486a6e53f434", "ResolvConfPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/resolv.conf", "HostnamePath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/hostname", "HostsPath": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/etc-hosts", "LogPath": "/var/lib/docker/containers/62cc3aeef11bbde0466ec03c2e1d3fe2e0b0380b732bea915a71954a263342fb/62cc3aeef11bbde0466ec03c2e1d3fe2e0b0380b732bea915a71954a263342fb-json.log", "Name": "/k8s_nginx_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_4", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/volumes/kubernetes.io~secret/default-token-dmx9w:/var/run/secrets/kubernetes.io/serviceaccount:ro,rslave", "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/etc-hosts:/etc/hosts", "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/containers/nginx/02a3148d:/dev/termination-log" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "container:2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c", "PortBindings": null, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "container:2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c", "Cgroup": "", "Links": null, "OomScoreAdj": 1000, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "seccomp=unconfined" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 2, "Memory": 0, "NanoCpus": 0, "CgroupParent": "cgroupfs.slice", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/asound", "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23-init/diff:/var/lib/docker/overlay2/4427ae02f52db721144b352ce214510061000c68e7786411e382212c9e619684/diff:/var/lib/docker/overlay2/73d0f6939e7114647ac4d448b901dc889ccb859af4ac027ae3223bd240ee5d3d/diff", "MergedDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23/merged", "UpperDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23/diff", "WorkDir": "/var/lib/docker/overlay2/71236732ee4a5853f496fbb43d9344fe9c4ae61259c9152546cfeed3b45c5b23/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/volumes/kubernetes.io~secret/default-token-dmx9w", "Destination": "/var/run/secrets/kubernetes.io/serviceaccount", "Mode": "ro,rslave", "RW": false, "Propagation": "rslave" }, { "Type": "bind", "Source": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/etc-hosts", "Destination": "/etc/hosts", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/edged/pods/53ccb836-4959-41cd-ac29-19eddc8aa0c5/containers/nginx/02a3148d", "Destination": "/dev/termination-log", "Mode": "", "RW": true, "Propagation": "rprivate" } ], "Config": { "Hostname": "nginx-edge", "Domainname": "", "User": "0", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "ExposedPorts": { "80/tcp": {} }, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NGINX_VERSION=1.16.1", "NJS_VERSION=0.3.8", "PKG_RELEASE=1" ], "Cmd": [ "nginx", "-g", "daemon off;" ], "Healthcheck": { "Test": [ "NONE" ] }, "ArgsEscaped": true, "Image": "sha256:5fad07aba15a1047c94b9818de93093310ed1bf2db98c185f5f8486a6e53f434", "Volumes": null, "WorkingDir": "", "Entrypoint": null, "OnBuild": null, "Labels": { "annotation.io.kubernetes.container.hash": "121b46b1", "annotation.io.kubernetes.container.restartCount": "4", "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy": "File", "annotation.io.kubernetes.pod.terminationGracePeriod": "30", "io.kubernetes.container.logpath": "/var/log/pods/default_nginx-rr9z6_53ccb836-4959-41cd-ac29-19eddc8aa0c5/nginx/4.log", "io.kubernetes.container.name": "nginx", "io.kubernetes.docker.type": "container", "io.kubernetes.pod.name": "nginx-rr9z6", "io.kubernetes.pod.namespace": "default", "io.kubernetes.pod.uid": "53ccb836-4959-41cd-ac29-19eddc8aa0c5", "io.kubernetes.sandbox.id": "2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c", "maintainer": "NGINX Docker Maintainers <docker-maint@nginx.com>" }, "StopSignal": "SIGTERM" }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": {} } } ] [root@edge-b305dc1eee21 ~]# docker inspect 2ba359afa65d [ { "Id": "2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c", "Created": "2020-02-13T02:31:10.641111343Z", "Path": "/pause", "Args": [], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 8490, "ExitCode": 0, "Error": "", "StartedAt": "2020-02-13T02:31:10.943041797Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e", "ResolvConfPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/resolv.conf", "HostnamePath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/hostname", "HostsPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/hosts", "LogPath": "/var/lib/docker/containers/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c/2ba359afa65dbdc7f806d1e50b1d618471049d79549635bcaa896021b68d1c8c-json.log", "Name": "/k8s_POD_nginx-rr9z6_default_53ccb836-4959-41cd-ac29-19eddc8aa0c5_1", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": null, "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "default", "PortBindings": {}, "RestartPolicy": { "Name": "", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "shareable", "Cgroup": "", "Links": null, "OomScoreAdj": -998, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": [ "seccomp=unconfined" ], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "ConsoleSize": [ 0, 0 ], "Isolation": "", "CpuShares": 2, "Memory": 0, "NanoCpus": 0, "CgroupParent": "cgroupfs.slice", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DiskQuota": 0, "KernelMemory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": 0, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": [ "/proc/acpi", "/proc/kcore", "/proc/keys", "/proc/latency_stats", "/proc/timer_list", "/proc/timer_stats", "/proc/sched_debug", "/proc/scsi", "/sys/firmware" ], "ReadonlyPaths": [ "/proc/asound", "/proc/bus", "/proc/fs", "/proc/irq", "/proc/sys", "/proc/sysrq-trigger" ] }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9-init/diff:/var/lib/docker/overlay2/07e6032dd42ab36194553def71e1c675ab62fc21cd1ec184a33e8d34efa879ba/diff", "MergedDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9/merged", "UpperDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9/diff", "WorkDir": "/var/lib/docker/overlay2/eb023f24e85aa0866da35ecbd64056a4ebeaf76c07d132f27e8430552569eeb9/work" }, "Name": "overlay2" }, "Mounts": [], "Config": { "Hostname": "nginx-edge", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ], "Cmd": null, "Image": "kubeedge/pause:3.1", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "/pause" ], "OnBuild": null, "Labels": { "app": "nginx", "controller-revision-hash": "557488dfcb", "io.kubernetes.container.name": "POD", "io.kubernetes.docker.type": "podsandbox", "io.kubernetes.pod.name": "nginx-rr9z6", "io.kubernetes.pod.namespace": "default", "io.kubernetes.pod.uid": "53ccb836-4959-41cd-ac29-19eddc8aa0c5", "pod-template-generation": "1" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "aa07ae52799da38a3e36819f3704f74b11256d7b182269c4c43018491fcbb7e2", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "/var/run/docker/netns/aa07ae52799d", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "3662c7489c804530f8f97857f759de00b1a6555a48eeb455ebe745a27ab752ce", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:02", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "NetworkID": "6ce2b7024fed591c6efde888dc4573e83774924f55c81035a89b6fefa9086c45", "EndpointID": "3662c7489c804530f8f97857f759de00b1a6555a48eeb455ebe745a27ab752ce", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "MacAddress": "02:42:ac:11:00:02", "DriverOpts": null } } } } ] ```

kerem

2026-02-26 04:33:58 +03:00

closed this issue
added the
triage
label

kerem commented

2026-02-26 04:33:59 +03:00

Author

Owner

@mageddo commented on GitHub (Mar 4, 2023):

Sorry I'm late. Unfortunately there were some reasons to this issue happen on DPS 2.x, DPS 3 fixes a lot of that, maybe all of them?! You can give a try at latest DPS version if you still need it and reopen the issue if still have problems, thanks.

@mageddo commented on GitHub (Mar 4, 2023): Sorry I'm late. Unfortunately there were some reasons to this issue happen on DPS 2.x, DPS 3 fixes a lot of that, maybe all of them?! You can give a try at latest DPS version if you still need it and reopen the issue if still have problems, thanks.