[PR #1027] [MERGED] Support $INCLUDE for master files #1884

New issue

Closed

opened 2026-03-16 02:52:36 +03:00 by kerem · 0 comments

kerem commented

2026-03-16 02:52:36 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/hickory-dns/hickory-dns/pull/1027
Author: @kachayev
Created: 2/23/2020
Status: ✅ Merged
Merged: 3/9/2020
Merged by: @bluejekyll

Base: master ← Head: feature-master-include

📝 Commits (10+)

93b916e Add $INCLUDE, make sure that tests fail now
9184e49 Update FileAuthority test to try to find subdomain mentioned in INCLUDE-d file
a02b57a Quick but working implementation, a lot of TODOs
1cc5e70 File reader to rely on Lexer to detect INCLUDE instead of manual processing
5feaeee Parser to notify about expectation for INCLUDE to be inlined before parsing
c390198 Differentiate absolute file path and relative file path when performing INCLUDE
df4a4fd Add comment about INCLUDE with custom origin
90dbcbb Merge branch 'master' into feature-master-include
b78d712 Extend logging for included files
a5b7da9 Track recursion depth and about in case of overlimit

📊 Changes

4 files changed (+151 additions, -24 deletions)

View changed files

📝 crates/client/src/serialize/txt/master.rs (+5 -9)
📝 crates/server/src/store/file/authority.rs (+142 -14)
📝 tests/test-data/named_test_configs/example.com.zone (+3 -1)
➕ tests/test-data/named_test_configs/include.example.com.zone (+1 -0)

📄 Description

Note: this is WIP. Sharing earlier to discuss some architectural challenges to get full support for $INCLUDE.

Because of how Parser/Lexer are implemented right now, the approach that I took is to update FileAuthority loader to read file line-by-line, run each line through Lexer to detect INCLUDEs, recursively process included file and inline the result. I extended the test file with a single $INCLUDE directive to make sure that the approach works in general. It exposes a few interesting problems tho'...

Lexer is executed twice for the file
Parser logic is not very intuitive now as it requires file to be pre-processed before parsing. Error message that explains this makes it a little bit better, but overall still feels bad.
(this one is the most significant) Regarding the RFC1035 $INCLUDE might specify a different origin domain. Meaning that inclining content of the file is not enough, we should actually parse it with respect to origin.

I'm not sure what's the best way forward from here because all of these couples logic of file loader/parser/lexer pretty heavily. Somewhat simple solution would be to define file loaders as a Lexer that internally maintains stack of nested lexers flattening stream of tokens. Problem #3 remains unsolved in this case, as far as we need to propagate not only the token, but current origin as well.

@bluejekyll What do you think?

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/hickory-dns/hickory-dns/pull/1027 **Author:** [@kachayev](https://github.com/kachayev) **Created:** 2/23/2020 **Status:** ✅ Merged **Merged:** 3/9/2020 **Merged by:** [@bluejekyll](https://github.com/bluejekyll) **Base:** `master` ← **Head:** `feature-master-include` --- ### 📝 Commits (10+) - [`93b916e`](https://github.com/hickory-dns/hickory-dns/commit/93b916e4ca34adc1b01c04fa4d1fa9e175f2871a) Add $INCLUDE, make sure that tests fail now - [`9184e49`](https://github.com/hickory-dns/hickory-dns/commit/9184e490b7a3703486ab29e7ed5f67e2fab18941) Update FileAuthority test to try to find subdomain mentioned in INCLUDE-d file - [`a02b57a`](https://github.com/hickory-dns/hickory-dns/commit/a02b57a3b638211655aa5eb5a82ac0195e48859f) Quick but working implementation, a lot of TODOs - [`1cc5e70`](https://github.com/hickory-dns/hickory-dns/commit/1cc5e70a52dc8a5480f79f0588a6b1ab197e7ea5) File reader to rely on Lexer to detect INCLUDE instead of manual processing - [`5feaeee`](https://github.com/hickory-dns/hickory-dns/commit/5feaeee1e36a0b3877eb898766bae5042b38c335) Parser to notify about expectation for INCLUDE to be inlined *before* parsing - [`c390198`](https://github.com/hickory-dns/hickory-dns/commit/c3901982c3c042a9836caafc90ba4ae8b0eec20c) Differentiate absolute file path and relative file path when performing INCLUDE - [`df4a4fd`](https://github.com/hickory-dns/hickory-dns/commit/df4a4fd0af7464a1db6927bcd0d40d786e5ee2cb) Add comment about INCLUDE with custom origin - [`90dbcbb`](https://github.com/hickory-dns/hickory-dns/commit/90dbcbb57490100affd099ddfe219c41a08ed860) Merge branch 'master' into feature-master-include - [`b78d712`](https://github.com/hickory-dns/hickory-dns/commit/b78d71261dd1a62d3464780ed3167d5f6062fbf6) Extend logging for included files - [`a5b7da9`](https://github.com/hickory-dns/hickory-dns/commit/a5b7da9031b0c810e3851ffc92c6fc94d9bdf6d8) Track recursion depth and about in case of overlimit ### 📊 Changes **4 files changed** (+151 additions, -24 deletions) <details> <summary>View changed files</summary> 📝 `crates/client/src/serialize/txt/master.rs` (+5 -9) 📝 `crates/server/src/store/file/authority.rs` (+142 -14) 📝 `tests/test-data/named_test_configs/example.com.zone` (+3 -1) ➕ `tests/test-data/named_test_configs/include.example.com.zone` (+1 -0) </details> ### 📄 Description Note: this is WIP. Sharing earlier to discuss some architectural challenges to get full support for `$INCLUDE`. Because of how `Parser`/`Lexer` are implemented right now, the approach that I took is to update `FileAuthority` loader to read file line-by-line, run each line through Lexer to detect INCLUDEs, recursively process included file and inline the result. I extended the test file with a single `$INCLUDE` directive to make sure that the approach works in general. It exposes a few interesting problems tho'... 1. `Lexer` is executed twice for the file 2. `Parser` logic is not very intuitive now as it requires file to be pre-processed before parsing. Error message that explains this makes it a little bit better, but overall still feels bad. 3. (this one is the most significant) Regarding the RFC1035 `$INCLUDE` might specify a different origin domain. Meaning that inclining content of the file is not enough, we should actually parse it with respect to origin. I'm not sure what's the best way forward from here because all of these couples logic of file loader/parser/lexer pretty heavily. Somewhat simple solution would be to define file loaders as a `Lexer` that internally maintains stack of nested lexers flattening stream of tokens. Problem #3 remains unsolved in this case, as far as we need to propagate not only the token, but current origin as well. @bluejekyll What do you think? --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>