[PR #1671] abx-plugin-title: use CURL_USER_AGENT when downloading page #2971

New issue

Closed

opened 2026-03-01 18:01:17 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 18:01:17 +03:00

Owner

Original Pull Request: https://github.com/ArchiveBox/ArchiveBox/pull/1671

State: closed
Merged: No

abx-plugin-title fetches the title with whatever works first:

reuse already downloaded page with dom/singlepage/wget
download the page with python-requests

the plugin documents/returns a curl command-line but it's never used. See https://github.com/ArchiveBox/ArchiveBox/issues/1670

At least we could mimick curl behaviour when downloading the page with python-requests by using the CURL_USER_AGENT setting.

Summary

Changes these areas

Bugfixes
Feature behavior
Command line interface
Configuration options
Internal architecture
Snapshot data layout on disk

**Original Pull Request:** https://github.com/ArchiveBox/ArchiveBox/pull/1671 **State:** closed **Merged:** No --- abx-plugin-title fetches the title with whatever works first: - reuse already downloaded page with dom/singlepage/wget - download the page with python-requests the plugin documents/returns a curl command-line but it's never used. See https://github.com/ArchiveBox/ArchiveBox/issues/1670 At least we could mimick curl behaviour when downloading the page with python-requests by using the CURL_USER_AGENT setting.  # Summary  # Related issues  # Changes these areas - [x] Bugfixes - [ ] Feature behavior - [ ] Command line interface - [ ] Configuration options - [ ] Internal architecture - [ ] Snapshot data layout on disk

kerem

2026-03-01 18:01:17 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#2971

No description provided.

Rows
Columns