[PR #1748] [MERGED] Add DNS traffic recorder plugin #4511

New issue

Closed

opened 2026-03-15 01:48:30 +03:00 by kerem · 0 comments

kerem commented

2026-03-15 01:48:30 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/ArchiveBox/ArchiveBox/pull/1748
Author: @pirate
Created: 12/31/2025
Status: ✅ Merged
Merged: 12/31/2025
Merged by: @pirate

Base: dev ← Head: claude/dns-traffic-recorder-plugin-dNbxC

📝 Commits (6)

13148fd Add DNS traffic recorder plugin
f2c20f1 Refactor dns plugin to use chrome_utils.js
5d8c93e Consolidate CDP connection logic into chrome_utils.js
47d9874 Merge remote-tracking branch 'origin/dev' into claude/dns-traffic-recorder-plugin-dNbxC
2a68248 Update all Chrome plugins to use shared chrome_utils.js
4839293 Fix test assertions to fail properly and add NXDOMAIN deduplication

📊 Changes

18 files changed (+616 additions, -748 deletions)

View changed files

📝 archivebox/plugins/chrome/chrome_utils.js (+174 -0)
📝 archivebox/plugins/consolelog/on_Snapshot__21_consolelog.bg.js (+23 -118)
➕ archivebox/plugins/dns/config.json (+21 -0)
➕ archivebox/plugins/dns/on_Snapshot__22_dns.bg.js (+240 -0)
➕ archivebox/plugins/dns/templates/icon.html (+0 -0)
➕ archivebox/plugins/dns/templates/thumbnail.html (+0 -0)
📝 archivebox/plugins/dom/on_Snapshot__53_dom.js (+4 -23)
📝 archivebox/plugins/headers/on_Snapshot__55_headers.js (+9 -29)
📝 archivebox/plugins/modalcloser/on_Snapshot__15_modalcloser.bg.js (+14 -46)
📝 archivebox/plugins/pdf/on_Snapshot__52_pdf.js (+4 -23)
📝 archivebox/plugins/pip/tests/test_pip_provider.py (+4 -1)
📝 archivebox/plugins/redirects/on_Snapshot__31_redirects.bg.js (+27 -114)
📝 archivebox/plugins/responses/on_Snapshot__24_responses.bg.js (+24 -123)
📝 archivebox/plugins/screenshot/on_Snapshot__51_screenshot.js (+4 -23)
📝 archivebox/plugins/seo/tests/test_seo.py (+6 -5)
📝 archivebox/plugins/ssl/on_Snapshot__23_ssl.bg.js (+22 -115)
📝 archivebox/plugins/ssl/tests/test_ssl.py (+14 -11)
📝 archivebox/plugins/staticfile/on_Snapshot__31_staticfile.bg.js (+26 -117)

📄 Description

Records hostname → IP resolutions during page load using Chrome CDP.
Uses Network.responseReceived events to capture DNS resolution data
and writes one JSON line per record to dns.jsonl.

Features:

Captures hostname to IP address mappings (A/AAAA records)
Records failed DNS lookups (NXDOMAIN)
Deduplicates resolution records per page load
Integrates with existing Chrome plugin infrastructure

Summary by cubic

Adds a DNS traffic recorder plugin that logs hostname → IP resolutions during page load via Chrome CDP and writes to dns.jsonl. Also refactors Chrome CDP-based plugins to use shared utilities, reducing duplicate code and standardizing navigation waits.

New Features
- Captures A/AAAA records and NXDOMAIN failures.
- Deduplicates hostname → IP mappings per page load.
- Configurable via DNS_ENABLED and DNS_TIMEOUT.
- Integrates with existing Chrome plugin infrastructure.
Migration
- SSL output is now ssl.jsonl (was ssl.json).
- Hooks no longer write listener.pid files.
- DNS plugin is enabled by default; control with DNS_ENABLED and DNS_TIMEOUT.

^{Written for commit 483929391d. Summary will update on new commits.}

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/ArchiveBox/ArchiveBox/pull/1748 **Author:** [@pirate](https://github.com/pirate) **Created:** 12/31/2025 **Status:** ✅ Merged **Merged:** 12/31/2025 **Merged by:** [@pirate](https://github.com/pirate) **Base:** `dev` ← **Head:** `claude/dns-traffic-recorder-plugin-dNbxC` --- ### 📝 Commits (6) - [`13148fd`](https://github.com/ArchiveBox/ArchiveBox/commit/13148fd6b510814402b68d25199deb39bdb4aa63) Add DNS traffic recorder plugin - [`f2c20f1`](https://github.com/ArchiveBox/ArchiveBox/commit/f2c20f141c3013b2107ee5ff680a3da5810c1faf) Refactor dns plugin to use chrome_utils.js - [`5d8c93e`](https://github.com/ArchiveBox/ArchiveBox/commit/5d8c93eaf492fa27d1937e973dc8784e25e3ae5f) Consolidate CDP connection logic into chrome_utils.js - [`47d9874`](https://github.com/ArchiveBox/ArchiveBox/commit/47d9874c1fdf9fc3c3a89911c3b39595cffde507) Merge remote-tracking branch 'origin/dev' into claude/dns-traffic-recorder-plugin-dNbxC - [`2a68248`](https://github.com/ArchiveBox/ArchiveBox/commit/2a6824860242bae049b640428bc1aca086910944) Update all Chrome plugins to use shared chrome_utils.js - [`4839293`](https://github.com/ArchiveBox/ArchiveBox/commit/483929391d16418a890b5d7fcc9b12971eea4edc) Fix test assertions to fail properly and add NXDOMAIN deduplication ### 📊 Changes **18 files changed** (+616 additions, -748 deletions) <details> <summary>View changed files</summary> 📝 `archivebox/plugins/chrome/chrome_utils.js` (+174 -0) 📝 `archivebox/plugins/consolelog/on_Snapshot__21_consolelog.bg.js` (+23 -118) ➕ `archivebox/plugins/dns/config.json` (+21 -0) ➕ `archivebox/plugins/dns/on_Snapshot__22_dns.bg.js` (+240 -0) ➕ `archivebox/plugins/dns/templates/icon.html` (+0 -0) ➕ `archivebox/plugins/dns/templates/thumbnail.html` (+0 -0) 📝 `archivebox/plugins/dom/on_Snapshot__53_dom.js` (+4 -23) 📝 `archivebox/plugins/headers/on_Snapshot__55_headers.js` (+9 -29) 📝 `archivebox/plugins/modalcloser/on_Snapshot__15_modalcloser.bg.js` (+14 -46) 📝 `archivebox/plugins/pdf/on_Snapshot__52_pdf.js` (+4 -23) 📝 `archivebox/plugins/pip/tests/test_pip_provider.py` (+4 -1) 📝 `archivebox/plugins/redirects/on_Snapshot__31_redirects.bg.js` (+27 -114) 📝 `archivebox/plugins/responses/on_Snapshot__24_responses.bg.js` (+24 -123) 📝 `archivebox/plugins/screenshot/on_Snapshot__51_screenshot.js` (+4 -23) 📝 `archivebox/plugins/seo/tests/test_seo.py` (+6 -5) 📝 `archivebox/plugins/ssl/on_Snapshot__23_ssl.bg.js` (+22 -115) 📝 `archivebox/plugins/ssl/tests/test_ssl.py` (+14 -11) 📝 `archivebox/plugins/staticfile/on_Snapshot__31_staticfile.bg.js` (+26 -117) </details> ### 📄 Description Records hostname → IP resolutions during page load using Chrome CDP. Uses Network.responseReceived events to capture DNS resolution data and writes one JSON line per record to dns.jsonl. Features: - Captures hostname to IP address mappings (A/AAAA records) - Records failed DNS lookups (NXDOMAIN) - Deduplicates resolution records per page load - Integrates with existing Chrome plugin infrastructure  --- ## Summary by cubic Adds a DNS traffic recorder plugin that logs hostname → IP resolutions during page load via Chrome CDP and writes to dns.jsonl. Also refactors Chrome CDP-based plugins to use shared utilities, reducing duplicate code and standardizing navigation waits. - **New Features** - Captures A/AAAA records and NXDOMAIN failures. - Deduplicates hostname → IP mappings per page load. - Configurable via DNS_ENABLED and DNS_TIMEOUT. - Integrates with existing Chrome plugin infrastructure. - **Migration** - SSL output is now ssl.jsonl (was ssl.json). - Hooks no longer write listener.pid files. - DNS plugin is enabled by default; control with DNS_ENABLED and DNS_TIMEOUT. <sup>Written for commit 483929391d16418a890b5d7fcc9b12971eea4edc. Summary will update on new commits.</sup>  --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-15 01:48:30 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#4511

No description provided.

Rows
Columns