[PR #1205] Fix hyphen placement in util.URL_REGEX #2862

New issue

Closed

opened 2026-03-01 18:00:58 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 18:00:58 +03:00

Owner

Original Pull Request: https://github.com/ArchiveBox/ArchiveBox/pull/1205

State: closed
Merged: Yes

Summary

Incorrect hyphen placement in URL_REGEX was allowing it to match more characters than intended. In a regex character class, a literal hyphen can only appear as the first character in the class, or it will be interpreted as the delimiter of a range of characters.

The issue fixed here caused the range of characters from [$-_] be treated as valid URL characters, instead of the intended set of three characters [-_$]. The incorrect range interpretation inadvertantly included most ASCII punctuation, most importantly the angle brackets, square brackets, and single quote that the expression uses to mark the end of a match.

This causes the expression to match a URL that has a "hostname" portion beginning with one of the intended "stop parsing" characters. For example:

https://<b>www</b>.example.com/  # MATCHES but should not
https://[for example]            # MATCHES but should not
scheme='https://'                # MATCHES, including final quote, but should not

Some test cases have been added to the URL_REGEX assert in archivebox.parsers to cover this possibility.

There are other URL_REGEX issues (#235, #287, #864, #874), but none that this change directly impacts.

Changes these areas

Bugfixes
Feature behavior
Command line interface
Configuration options
Internal architecture
Snapshot data layout on disk

**Original Pull Request:** https://github.com/ArchiveBox/ArchiveBox/pull/1205 **State:** closed **Merged:** Yes --- # Summary Incorrect hyphen placement in `URL_REGEX` was allowing it to match more characters than intended. In a regex character class, a literal hyphen can only appear as the first character in the class, or it will be interpreted as the delimiter of a range of characters. The issue fixed here caused the range of characters from `[$-_]` be treated as valid URL characters, instead of the intended set of three characters `[-_$]`. The incorrect range interpretation inadvertantly included most ASCII punctuation, most importantly the angle brackets, square brackets, and single quote that the expression uses to mark the end of a match. This causes the expression to match a URL that has a "hostname" portion beginning with one of the intended "stop parsing" characters. For example: ``` https://<b>www</b>.example.com/ # MATCHES but should not https://[for example] # MATCHES but should not scheme='https://' # MATCHES, including final quote, but should not ``` Some test cases have been added to the `URL_REGEX` assert in archivebox.parsers to cover this possibility. # Related issues There are other `URL_REGEX` issues (#235, #287, #864, #874), but none that this change directly impacts. # Changes these areas - [X] Bugfixes - [ ] Feature behavior - [ ] Command line interface - [ ] Configuration options - [ ] Internal architecture - [ ] Snapshot data layout on disk

kerem

2026-03-01 18:00:58 +03:00

closed this issue
added the
pull-request
label

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#2862

No description provided.

Rows
Columns