[PR #2385] [MERGED] [RFC] (temporarily) add tests that rely on public DNS infrastructure #3003

New issue

Closed

opened 2026-03-16 11:19:51 +03:00 by kerem · 0 comments

kerem commented

2026-03-16 11:19:51 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hickory-dns/hickory-dns/pull/2385
Author: @japaric
Created: 8/26/2024
Status: ✅ Merged
Merged: 8/30/2024
Merged by: @djc

Base: main ← Head: ja-test-against-ede-dot-com

📝 Commits (2)

dc44a69 dns-test: allow testing against the public DNS infra
df38ce6 add ede-dot-com test suite

📊 Changes

10 files changed (+1218 additions, -7 deletions)

View changed files

📝 .github/workflows/conformance.yml (+3 -0)
📝 conformance/packages/dns-test/src/container/network.rs (+13 -6)
📝 conformance/packages/dns-test/src/trust_anchor.rs (+22 -1)
📝 conformance/packages/dns-test/src/zone_file/mod.rs (+8 -0)
📝 justfile (+30 -0)
➕ tests/ede-dot-com/Cargo.lock (+703 -0)
➕ tests/ede-dot-com/Cargo.toml (+12 -0)
➕ tests/ede-dot-com/README.md (+5 -0)
➕ tests/ede-dot-com/src/lib.rs (+395 -0)
➕ tests/ede-dot-com/src/sanity_check.rs (+27 -0)

📄 Description

Background

Recently, @justahero found this great resource: https://extended-dns-errors.com/

Most of its 63 subdomains are made of incorrectly configured authoritative name servers with the goal of serving as a testbed for the implementation of RFC8914 (AKA Extended DNS Errors AKA EDE)

I originally thought it would be great to port the test cases under that domain into conformance-tests to serve as tests for the (not yet implemented) EDE feature but after some local testing against the subdomains they uncovered what seem to be DNSSEC bugs in hickory.

Proposal

I would like to add tests that hit these 63 subdomains for the purpose of local testing. The new tests will live in the public-dns-tests package.

These tests will serve as a TODO list of failures that need to be investigated. Once a test failure has been confirmed to be a DNSSEC bug, the test will be rewritten to not rely on public DNS resources (i.e. to use the local / private dns_test::{Resolver,Nameserver} containers) and moved into conformance-tests (or into e2e-tests if necessary).

Once all the observed test failures have been investigated and fixed, the public-dns-tests package can be removed.

Alternatively, we could walk the extra mile and port all the tests / subdomains, including the ones that pass today, and only after that remove the package.

Details

Because this test suite is for local development, the tests won't run in CI. CI will only check that the test code compiles. The rationale for this is that tests that rely on internet access are inherently flaky so I'd rather not add spurious failures to the CI checks. People running the tests locally can determine if a test is flaky and re-run it if needed.

The tests are set up to spin up local unbound and hickory resolvers configured with DNSSEC enabled and root hints and trust anchors that work with the public DNS infra. A query about an incorrectly configured subdomain is sent to both local resolvers and the test checks that both resolvers produce the same answer. What's being check is that the RCODE matches (e.g. NOERROR or SERVFAIL) and that the flags match (the Authenticated Data one is particularly important). To focus on DNSSEC bugs, the EDE codes in the responses are not compared; also because that would make a lot more tests fail because hickory does not implement EDE yet

The tests are set up using a tests! macro in public-dns-tests that indicates the "state" (or I guess category) of the test. A PASS(ing) test serves as a regression test and it's expected to pass. A FIXME test is a known failure that needs to be investigated and fixed. PORTED is for tests that have been ported to conformance-tests. The first test to be added to this category is going to be the one for the unsigned. domain (PR #2380)

PS 29 of the 63 tests are in the FIXME category. There seem to be some common themes (e.g. lack of KSK or DS validation) that may indicate that the same bug is triggering multiple test failures.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hickory-dns/hickory-dns/pull/2385 **Author:** [@japaric](https://github.com/japaric) **Created:** 8/26/2024 **Status:** ✅ Merged **Merged:** 8/30/2024 **Merged by:** [@djc](https://github.com/djc) **Base:** `main` ← **Head:** `ja-test-against-ede-dot-com` --- ### 📝 Commits (2) - [`dc44a69`](https://github.com/hickory-dns/hickory-dns/commit/dc44a6919b88f515bfa6155b774659ae947fb899) dns-test: allow testing against the public DNS infra - [`df38ce6`](https://github.com/hickory-dns/hickory-dns/commit/df38ce6a8839b1c670d01cbf0d2ca42fd3b4308c) add ede-dot-com test suite ### 📊 Changes **10 files changed** (+1218 additions, -7 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/conformance.yml` (+3 -0) 📝 `conformance/packages/dns-test/src/container/network.rs` (+13 -6) 📝 `conformance/packages/dns-test/src/trust_anchor.rs` (+22 -1) 📝 `conformance/packages/dns-test/src/zone_file/mod.rs` (+8 -0) 📝 `justfile` (+30 -0) ➕ `tests/ede-dot-com/Cargo.lock` (+703 -0) ➕ `tests/ede-dot-com/Cargo.toml` (+12 -0) ➕ `tests/ede-dot-com/README.md` (+5 -0) ➕ `tests/ede-dot-com/src/lib.rs` (+395 -0) ➕ `tests/ede-dot-com/src/sanity_check.rs` (+27 -0) </details> ### 📄 Description # Background Recently, @justahero found this great resource: https://extended-dns-errors.com/ Most of its 63 subdomains are made of incorrectly configured authoritative name servers with the goal of serving as a testbed for the implementation of RFC8914 (AKA Extended DNS Errors AKA EDE) I originally thought it would be great to port the test cases under that domain into `conformance-tests` to serve as tests for the (not yet implemented) EDE feature but after some local testing against the subdomains they uncovered what seem to be DNSSEC bugs in hickory. # Proposal I would like to add tests that hit these 63 subdomains for the purpose of *local testing*. The new tests will live in the `public-dns-tests` package. These tests will serve as a TODO list of failures that need to be investigated. Once a test failure has been confirmed to be a DNSSEC bug, the test will be rewritten to not rely on public DNS resources (i.e. to use the local / private `dns_test::{Resolver,Nameserver}` containers) and moved into `conformance-tests` (or into `e2e-tests` if necessary). Once all the observed test failures have been investigated and fixed, the `public-dns-tests` package can be removed. Alternatively, we could walk the extra mile and port all the tests / subdomains, including the ones that pass today, and only after that remove the package. # Details Because this test suite is for local development, the tests won't run in CI. CI will only check that the test code compiles. The rationale for this is that tests that rely on internet access are inherently flaky so I'd rather not add spurious failures to the CI checks. People running the tests locally can determine if a test is flaky and re-run it if needed. The tests are set up to spin up local unbound and hickory resolvers configured with DNSSEC enabled and root hints and trust anchors that work with the public DNS infra. A query about an incorrectly configured subdomain is sent to both local resolvers and the test checks that both resolvers produce the same answer. What's being check is that the RCODE matches (e.g. NOERROR or SERVFAIL) and that the flags match (the Authenticated Data one is particularly important). To focus on DNSSEC bugs, the EDE codes in the responses are not compared; also because that would make a lot more tests fail because hickory does not implement EDE yet The tests are set up using a `tests!` macro in `public-dns-tests` that indicates the "state" (or I guess category) of the test. A PASS(ing) test serves as a regression test and it's expected to pass. A FIXME test is a known failure that needs to be investigated and fixed. PORTED is for tests that have been ported to `conformance-tests`. The first test to be added to this category is going to be the one for the `unsigned.` domain (PR #2380) PS 29 of the 63 tests are in the FIXME category. There seem to be some common themes (e.g. lack of KSK or DS validation) that may indicate that the same bug is triggering multiple test failures. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>