[PR #1751] Clean up on_Crawl hooks and remove dead code #3010

New issue

Closed

opened 2026-03-01 18:01:23 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 18:01:23 +03:00

Owner

Original Pull Request: https://github.com/ArchiveBox/ArchiveBox/pull/1751

State: closed
Merged: Yes

Deleted dead/duplicate hooks:

wget/on_Crawl__10_install_wget.py (duplicate of __10_wget_validate_config.py)
chrome/on_Crawl__00_chrome_install.py (simpler version, kept full one)
chrome/on_Crawl__20_chrome_launch.bg.js (legacy, kept __30 version)
singlefile/on_Crawl__20_install_singlefile_extension.js (disabled/dead)
istilldontcareaboutcookies/on_Crawl__20_install_*.js (legacy)
ublock/on_Crawl__03_ublock.js (legacy, kept __20 version)
Entire captcha2/ plugin (legacy version of twocaptcha/)

Renamed hooks to follow consistent pattern: on_Crawl__XX__.
Priority bands:
00-09: Binary/extension installation 10-19: Config validation 20-29: Browser launch and post-launch config

Final hooks:
00 ripgrep_install.py, 01 chrome_install.py 02 istilldontcareaboutcookies_install.js 03 ublock_install.js, 04 singlefile_install.js 05 twocaptcha_install.js 10 chrome_validate.py, 11 wget_validate.py 20 chrome_launch.bg.js, 25 twocaptcha_config.js

Summary

Changes these areas

Bugfixes
Feature behavior
Command line interface
Configuration options
Internal architecture
Snapshot data layout on disk

Summary by cubic

Cleaned up Crawl-level hooks by removing legacy/duplicate code and standardizing hook names and priorities. Chrome launch is now a single, updated hook with better extension detection and cleaner outputs.

Refactors
- Removed dead hooks (legacy chrome install/launch, singlefile extension, old ublock/cookies scripts, duplicate wget validate) and the legacy captcha2 plugin in favor of twocaptcha.
- Renamed hooks to on_Crawl__XX__ with priority bands: 00-09 install, 10-19 validate, 20-29 launch/config.
- Consolidated Chrome launch into on_Crawl__20_chrome_launch.bg.js; writes outputs to the current dir, resolves real extension IDs via chrome://extensions, and records extensions.json after verification.
Migration
- If you used captcha2, switch to the twocaptcha hooks (on_Crawl__05_twocaptcha_install.js and on_Crawl__25_twocaptcha_config.js).
- Update any docs/scripts that reference old hook filenames.

^{Written for commit 4c77949197. Summary will update on new commits.}

**Original Pull Request:** https://github.com/ArchiveBox/ArchiveBox/pull/1751 **State:** closed **Merged:** Yes --- Deleted dead/duplicate hooks: - wget/on_Crawl__10_install_wget.py (duplicate of __10_wget_validate_config.py) - chrome/on_Crawl__00_chrome_install.py (simpler version, kept full one) - chrome/on_Crawl__20_chrome_launch.bg.js (legacy, kept __30 version) - singlefile/on_Crawl__20_install_singlefile_extension.js (disabled/dead) - istilldontcareaboutcookies/on_Crawl__20_install_*.js (legacy) - ublock/on_Crawl__03_ublock.js (legacy, kept __20 version) - Entire captcha2/ plugin (legacy version of twocaptcha/) Renamed hooks to follow consistent pattern: on_Crawl__XX_<plugin>_<action>.<ext> Priority bands: 00-09: Binary/extension installation 10-19: Config validation 20-29: Browser launch and post-launch config Final hooks: 00 ripgrep_install.py, 01 chrome_install.py 02 istilldontcareaboutcookies_install.js 03 ublock_install.js, 04 singlefile_install.js 05 twocaptcha_install.js 10 chrome_validate.py, 11 wget_validate.py 20 chrome_launch.bg.js, 25 twocaptcha_config.js  # Summary  # Related issues  # Changes these areas - [ ] Bugfixes - [ ] Feature behavior - [ ] Command line interface - [ ] Configuration options - [ ] Internal architecture - [ ] Snapshot data layout on disk  --- ## Summary by cubic Cleaned up Crawl-level hooks by removing legacy/duplicate code and standardizing hook names and priorities. Chrome launch is now a single, updated hook with better extension detection and cleaner outputs. - **Refactors** - Removed dead hooks (legacy chrome install/launch, singlefile extension, old ublock/cookies scripts, duplicate wget validate) and the legacy captcha2 plugin in favor of twocaptcha. - Renamed hooks to on_Crawl__XX_<plugin>_<action> with priority bands: 00-09 install, 10-19 validate, 20-29 launch/config. - Consolidated Chrome launch into on_Crawl__20_chrome_launch.bg.js; writes outputs to the current dir, resolves real extension IDs via chrome://extensions, and records extensions.json after verification. - **Migration** - If you used captcha2, switch to the twocaptcha hooks (on_Crawl__05_twocaptcha_install.js and on_Crawl__25_twocaptcha_config.js). - Update any docs/scripts that reference old hook filenames. <sup>Written for commit 4c77949197cd2481e0ff48df083263dc0b9cb8ae. Summary will update on new commits.</sup>

kerem

2026-03-01 18:01:24 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#3010

No description provided.

Rows
Columns