mirror of
https://github.com/hickory-dns/hickory-dns.git
synced 2026-04-25 11:15:54 +03:00
[GH-ISSUE #3023] hickory-resolver hangs with 100% CPU when querying DNSSEC records via systemd-resolved #1113
Labels
No labels
blocked
breaking-change
bug
bug:critical
bug:tests
cleanup
compliance
compliance
compliance
crate:all
crate:client
crate:native-tls
crate:proto
crate:recursor
crate:resolver
crate:resolver
crate:rustls
crate:server
crate:util
dependencies
docs
duplicate
easy
easy
enhance
enhance
enhance
feature:dns-over-https
feature:dns-over-quic
feature:dns-over-tls
feature:dnsssec
feature:global_lb
feature:mdns
feature:tsig
features:edns
has workaround
ops
perf
platform:WASM
platform:android
platform:fuchsia
platform:linux
platform:macos
platform:windows
pull-request
question
test
tools
tools
trust
unclear
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/hickory-dns#1113
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @mdecimus on GitHub (May 31, 2025).
Original GitHub issue: https://github.com/hickory-dns/hickory-dns/issues/3023
Describe the bug
hickory-resolvergets stuck in (what seems to be) an infinite loop consuming 100% CPU when resolving DNSSEC-enabled records (such asmail.protonmail.ch) through systemd-resolved as the system resolver.I've been able to consistently reproduce this issue on two different servers running systemd versions
237and249. Interestingly, the same query works correctly on a third server running systemd version256, suggesting this is likely a bug in older versions of systemd-resolved.While this appears to be an upstream systemd issue that has been fixed in newer versions, many production servers are still running older systemd releases. Given the widespread deployment of these older versions, it would be to have a workaround or timeout mechanism in
hickory-resolverto handle this scenario gracefully rather than hanging indefinitely.I've attached a minimal Rust reproduction case that demonstrates the issue, along with a GDB stack trace showing where the process gets stuck. The reproduction code should trigger the hang reliably on systems with affected systemd-resolved versions.
Here are the systemd versions of both servers where I was able to reproduce the issue:
To Reproduce
Run the following code to reproduce:
System:
Version:
Crate: hickory-resolver
Version: 0.25.2
Additional context
@mdecimus commented on GitHub (May 31, 2025):
Stack trace:
@djc commented on GitHub (May 31, 2025):
Can you add a basic tracing resolver and run a failing case with RUST_LOG=trace? I don't have easy access to systemd-resolved.
@mdecimus commented on GitHub (May 31, 2025):
After enabling tracing, the following error is logged continuously:
@djc commented on GitHub (Jun 2, 2025):
So it looks like this is stuck recursing infinitely? @divergentdave you were working on this stuff in #2936, do you think some of that stuff can be (easily) backported? @mdecimus might be interesting to try with current main.
@mdecimus commented on GitHub (Jun 2, 2025):
Yes, although one user reported that DNSSEC queries were completing after 2 hours (I haven't confirmed this myself).
Sure, will try it in a few hours.
@mdecimus commented on GitHub (Jun 2, 2025):
@djc I can confirm that the issue is no longer present in the main branch (
v0.26.0-alpha.1). Also, the record is resolved much faster than inv0.25.2, even withsystemd-resolvedversion256(where DNSSEC resolution works) it was taking about 3 second and now it is practically immediate.Should I close this or would you like to backport the fixes?
And finally, do you have a time estimate of when
v0.26is going to be released? If it's not anytime soon I'll disable DNSSEC until a new version is out.@djc commented on GitHub (Jun 2, 2025):
Great to hear it works well on main!
I don't think we're planning on releasing 0.26.0 any time soon. We could release a 0.26.0-alpha.1 which you could potentially use, if that would work for you? Otherwise, we might be able to backport some of this work.
@mdecimus commented on GitHub (Jun 2, 2025):
Yes, that would be great, thank you!
Stalwart uses
hickory-resolvervia the mail-auth crate (which you've contributed to in the past as well!). If you believe the alpha version is stable enough for basic TXT, MX, and A/AAAA record resolution, I’d be happy to updatemail-authto depend on0.26.0-alpha.1.@djc commented on GitHub (Jun 2, 2025):
@divergentdave shall I just publish a release now, or do you think there's anything in flight we should wait on? Any risks from using current
main? Seems like a good option to me.@divergentdave commented on GitHub (Jun 2, 2025):
I don't have any concerns with cutting a release, but I expect we'll continue to have more breaking changes coming up soon.
@mdecimus commented on GitHub (Jun 3, 2025):
Hi @djc, sorry to bother you again with this issue but I was wondering if there’s a chance 0.26.0-alpha.1 could be released sometime this week?
There are daily reports from users experiencing high CPU usage, and the fix in the upcoming release would really help. If the alpha isn’t likely to be published soon, I’ll go ahead and disable DNSSEC when using the system resolver as a temporary workaround. Thanks in advance!
@djc commented on GitHub (Jun 3, 2025):
It's publishing now:
https://github.com/hickory-dns/hickory-dns/actions/runs/15413133716/job/43369595923
@mdecimus commented on GitHub (Jun 3, 2025):
Thank you!