[GH-ISSUE #1540] Client gets stuck in broken state (ProtoError "busy") after a few minutes #695

Closed
opened 2026-03-15 23:52:27 +03:00 by kerem · 3 comments
Owner

Originally created by @ErwanDL on GitHub (Aug 19, 2021).
Original GitHub issue: https://github.com/hickory-dns/hickory-dns/issues/1540

Hello, I am encountering a weird bug where my DNS client becomes broken and stays in that state after having worked normally for a few minutes. My use case is a background-running task, in which an AsyncClient queries my Kubernetes cluster's DNS (kube-dns) every 5 seconds to list the IPs of deployed pods in a service. When I start the task, it works perfectly fine for the first ~2 minutes, returning the correct results, and then the queries start returning an Err of type ProtoError { kind: Busy }. I can observe the exact same behaviour on my local machine and on my prod cluster.

Here are high-level logs of the requests first succeeding, then failing:

// This works
[2021-08-19T15:26:51Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:26:51Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
[2021-08-19T15:26:56Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:26:56Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
[2021-08-19T15:27:01Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:01Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:06Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:06Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:11Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:11Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:16Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:16Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:21Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:21Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
[2021-08-19T15:27:26Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:26Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
[2021-08-19T15:27:31Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:31Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:36Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:36Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
[2021-08-19T15:27:41Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:41Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:46Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:46Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:27:51Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:51Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
[2021-08-19T15:27:56Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:27:56Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:28:01Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:01Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:28:06Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:06Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}.
[2021-08-19T15:28:11Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:11Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}.
// Starts to fail
[2021-08-19T15:28:16Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:16Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:21Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:21Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:26Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:26Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:31Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:31Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:36Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:36Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:41Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:41Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:46Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:46Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:51Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:51Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:28:56Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:28:56Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:01Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:01Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:06Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:06Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:11Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:11Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:16Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:16Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:21Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:21Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:26Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:26Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }.
[2021-08-19T15:29:31Z DEBUG myapp] Querying DNS server...
[2021-08-19T15:29:31Z WARN  myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }

There is no other component in my application using the client, and I don't believe anything is interacting with it. Can you help me understand what this ProtoError { kind: Busy } means, and what could cause it to enter this broken state ?

Originally created by @ErwanDL on GitHub (Aug 19, 2021). Original GitHub issue: https://github.com/hickory-dns/hickory-dns/issues/1540 Hello, I am encountering a weird bug where my DNS client becomes broken and stays in that state after having worked normally for a few minutes. My use case is a background-running task, in which an `AsyncClient` queries my Kubernetes cluster's DNS (`kube-dns`) every 5 seconds to list the IPs of deployed pods in a service. When I start the task, it works perfectly fine for the first ~2 minutes, returning the correct results, and then the queries start returning an `Err` of type `ProtoError { kind: Busy }`. I can observe the exact same behaviour on my local machine and on my prod cluster. Here are high-level logs of the requests first succeeding, then failing: ``` // This works [2021-08-19T15:26:51Z DEBUG myapp] Querying DNS server... [2021-08-19T15:26:51Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. [2021-08-19T15:26:56Z DEBUG myapp] Querying DNS server... [2021-08-19T15:26:56Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. [2021-08-19T15:27:01Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:01Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:06Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:06Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:11Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:11Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:16Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:16Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:21Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:21Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. [2021-08-19T15:27:26Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:26Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. [2021-08-19T15:27:31Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:31Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:36Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:36Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. [2021-08-19T15:27:41Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:41Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:46Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:46Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:27:51Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:51Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. [2021-08-19T15:27:56Z DEBUG myapp] Querying DNS server... [2021-08-19T15:27:56Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:28:01Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:01Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:28:06Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:06Z DEBUG myapp] Backends returned by DNS server: {10.68.2.42, 10.68.1.74}. [2021-08-19T15:28:11Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:11Z DEBUG myapp] Backends returned by DNS server: {10.68.1.74, 10.68.2.42}. // Starts to fail [2021-08-19T15:28:16Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:16Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:21Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:21Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:26Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:26Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:31Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:31Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:36Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:36Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:41Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:41Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:46Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:46Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:51Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:51Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:28:56Z DEBUG myapp] Querying DNS server... [2021-08-19T15:28:56Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:01Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:01Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:06Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:06Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:11Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:11Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:16Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:16Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:21Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:21Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:26Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:26Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) }. [2021-08-19T15:29:31Z DEBUG myapp] Querying DNS server... [2021-08-19T15:29:31Z WARN myapp] DNS query failed, retrying in 5 seconds: Error { kind: Proto(ProtoError { kind: Busy }) } ``` There is no other component in my application using the client, and I don't believe anything is interacting with it. Can you help me understand what this `ProtoError { kind: Busy }` means, and what could cause it to enter this broken state ?
kerem closed this issue 2026-03-15 23:52:32 +03:00
Author
Owner

@djc commented on GitHub (Aug 25, 2021):

What version are you using? Maybe try the main branch? There have been some changes that might impact this.

<!-- gh-comment-id:905305241 --> @djc commented on GitHub (Aug 25, 2021): What version are you using? Maybe try the main branch? There have been some changes that might impact this.
Author
Owner

@ErwanDL commented on GitHub (Aug 25, 2021):

I have resorted to using tokio's lookup_host in the meantime instead, I unfortunately don't have the setup to reproduce this anymore.

<!-- gh-comment-id:905313480 --> @ErwanDL commented on GitHub (Aug 25, 2021): I have resorted to using tokio's [lookup_host](https://tokio-rs.github.io/tokio/doc/tokio/net/fn.lookup_host.html) in the meantime instead, I unfortunately don't have the setup to reproduce this anymore.
Author
Owner

@bluejekyll commented on GitHub (Aug 25, 2021):

I don't really recommend using the client for DNS query/resolution. The resolver is designed for that use case.

For this particular issue, there would be a lot of questions to pull apart. What protocol are you using for the connections? Are you using the DnsMultiplexer? (that will produce a busy error if there are futures that aren't being consumed, https://github.com/bluejekyll/trust-dns/blob/main/crates/proto/src/xfer/dns_multiplexer.rs#L274 )

My guess, based on that error, is that your code is opening more connections than are being consuming, and you're running into the default channel buffer size of 32.

<!-- gh-comment-id:905545398 --> @bluejekyll commented on GitHub (Aug 25, 2021): I don't really recommend using the client for DNS query/resolution. The resolver is designed for that use case. For this particular issue, there would be a lot of questions to pull apart. What protocol are you using for the connections? Are you using the DnsMultiplexer? (that will produce a busy error if there are futures that aren't being consumed, https://github.com/bluejekyll/trust-dns/blob/main/crates/proto/src/xfer/dns_multiplexer.rs#L274 ) My guess, based on that error, is that your code is opening more connections than are being consuming, and you're running into the default channel buffer size of 32.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/hickory-dns#695
No description provided.