starred/ArchiveBox

Fork

You've already forked ArchiveBox

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 17:16:00 +03:00

Code Issues 624 Projects Releases 5 Packages Wiki Activity

[PR #426] [MERGED] feat: Initial version of readability extractor #1161

New issue

Closed

opened 2026-03-01 14:48:40 +03:00 by kerem · 0 comments

kerem commented

2026-03-01 14:48:40 +03:00

Owner

Copy link

📋 Pull Request Information

Original PR: https://github.com/ArchiveBox/ArchiveBox/pull/426
Author: @cdvv7788
Created: 8/7/2020
Status: ✅ Merged
Merged: 8/14/2020
Merged by: @pirate

Base: master ← Head: readability-extractor

📝 Commits (10+)

7e2b249 feat: Initial version of readability extractor
b33c66a feat: Split output of readability into multiple files
61e08a7 docs: Update docs link
a147626 feat: Avoid running readability when the target is a file
0ec747f feat: Look in wget, singlefile or dom outputs before attempting to download the information again
dc87d8b tests: Update failing tests
8aa7b34 tests: Add readability to ignored methods in tests
2a68af1 tests: Add readability tests
5dc7e63 feat: Update dockerfile to support readability
4d44b17 tests: Add readability steps to CI

📊 Changes

9 files changed (+182 additions, -1 deletions)

View changed files

📝 Dockerfile (+2 -0)
📝 archivebox/config/__init__.py (+15 -1)
📝 archivebox/extractors/__init__.py (+3 -0)
➕ archivebox/extractors/readability.py (+113 -0)
📝 archivebox/index/schema.py (+2 -0)
📝 archivebox/themes/legacy/link_details.html (+12 -0)
📝 tests/fixtures.py (+1 -0)
📝 tests/test_extractors.py (+32 -0)
📝 tests/test_init.py (+2 -0)

📄 Description

Summary

Initial working version of the readability extractor, using https://github.com/cdvv7788/readability-extractor as wrapper (JSDOM, readability, dompurify).
This has a long way to go, but for now it is functional.

Questions:

Currently, it accepts the html directly as an arg. When something fails, it will output the whole command, which will have a HUGE html attached to it...is that acceptable? Is it better to save it to a tmp_file and load the contents from there?
The wrapper is pretty basic. It does what it needs at the bare minimum to work. Improvements there will definitely be necessary.
Currently, the output is saved in a file named readability.json inside of the archive. It has several fields. What is the output you are expecting? Is it ok to keep this json or should we extract an specific field?

**Related issues: #69

Changes these areas

Bugfixes
Feature behavior
Command line interface
Configuration options
Internal architecture
Archived data layout on disk

Pending work

Add tests
Add readability.json to the detail html index
Add support for readability to the Dockerfile
Add support for readability to the CI pipeline

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/ArchiveBox/ArchiveBox/pull/426 **Author:** [@cdvv7788](https://github.com/cdvv7788) **Created:** 8/7/2020 **Status:** ✅ Merged **Merged:** 8/14/2020 **Merged by:** [@pirate](https://github.com/pirate) **Base:** `master` ← **Head:** `readability-extractor` --- ### 📝 Commits (10+) - [`7e2b249`](https://github.com/ArchiveBox/ArchiveBox/commit/7e2b249388e4a530dcbcd374de469033eeb36c18) feat: Initial version of readability extractor - [`b33c66a`](https://github.com/ArchiveBox/ArchiveBox/commit/b33c66a9f77e973b0fa338038fcf18dc2eddb584) feat: Split output of readability into multiple files - [`61e08a7`](https://github.com/ArchiveBox/ArchiveBox/commit/61e08a7c430b634022088273ddb0a74a1f6e8b89) docs: Update docs link - [`a147626`](https://github.com/ArchiveBox/ArchiveBox/commit/a14762640e68d32dcb8aa29639fce7474c50a1d6) feat: Avoid running readability when the target is a file - [`0ec747f`](https://github.com/ArchiveBox/ArchiveBox/commit/0ec747f64e9b47fd08555d2c17b555874ace0a90) feat: Look in wget, singlefile or dom outputs before attempting to download the information again - [`dc87d8b`](https://github.com/ArchiveBox/ArchiveBox/commit/dc87d8b68c717438e95a713040fbcdd1849f8508) tests: Update failing tests - [`8aa7b34`](https://github.com/ArchiveBox/ArchiveBox/commit/8aa7b34de731e48e1aba84582a5df43e2add410b) tests: Add readability to ignored methods in tests - [`2a68af1`](https://github.com/ArchiveBox/ArchiveBox/commit/2a68af1b946c59a042d49c3bda414c39fee136b6) tests: Add readability tests - [`5dc7e63`](https://github.com/ArchiveBox/ArchiveBox/commit/5dc7e63792286c31988a964e5d5ef3a89a70ced8) feat: Update dockerfile to support readability - [`4d44b17`](https://github.com/ArchiveBox/ArchiveBox/commit/4d44b172e67e23fd6a2fb835b9d6547900701ff0) tests: Add readability steps to CI ### 📊 Changes **9 files changed** (+182 additions, -1 deletions) <details> <summary>View changed files</summary> 📝 `Dockerfile` (+2 -0) 📝 `archivebox/config/__init__.py` (+15 -1) 📝 `archivebox/extractors/__init__.py` (+3 -0) ➕ `archivebox/extractors/readability.py` (+113 -0) 📝 `archivebox/index/schema.py` (+2 -0) 📝 `archivebox/themes/legacy/link_details.html` (+12 -0) 📝 `tests/fixtures.py` (+1 -0) 📝 `tests/test_extractors.py` (+32 -0) 📝 `tests/test_init.py` (+2 -0) </details> ### 📄 Description # Summary Initial working version of the readability extractor, using https://github.com/cdvv7788/readability-extractor as wrapper (JSDOM, readability, dompurify). This has a long way to go, but for now it is functional. Questions: - Currently, it accepts the html directly as an arg. When something fails, it will output the whole command, which will have a HUGE html attached to it...is that acceptable? Is it better to save it to a tmp_file and load the contents from there? - The wrapper is pretty basic. It does what it needs at the bare minimum to work. Improvements there will definitely be necessary. - Currently, the output is saved in a file named `readability.json` inside of the archive. It has several fields. What is the output you are expecting? Is it ok to keep this json or should we extract an specific field? **Related issues: #69 # Changes these areas - [ ] Bugfixes - [X] Feature behavior - [ ] Command line interface - [ ] Configuration options - [ ] Internal architecture - [X] Archived data layout on disk # Pending work - [x] Add tests - [x] Add `readability.json` to the detail html index - [x] Add support for readability to the `Dockerfile` - [x] Add support for readability to the CI pipeline --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

kerem

2026-03-01 14:48:40 +03:00

closed this issue
added the
pull-request
label

kerem referenced this issue

2026-03-01 14:49:22 +03:00

[PR #1161] [CLOSED] v0.6.3 Release (Sourcery refactored) #1336

kerem referenced this issue

2026-03-01 18:00:56 +03:00

[PR #1161] v0.6.3 Release (Sourcery refactored) #2847

kerem referenced this issue

2026-03-15 01:39:51 +03:00

[PR #1161] [CLOSED] v0.6.3 Release (Sourcery refactored) #4351

No Branch/Tag specified

dev

main

abx-dl-refactor

claude/issue-1688-20251229-2233

claude/s3-support-migration-JZQcf

claude/review-snapshot-archive-RJOkr

claude/review-code-quality-macdO

claude/reduce-chrome-duplication-JgWVr

fix/chrome-args-comma-parsing

claude/issue-668-20251229-2145

claude/issue-1664-20251229-2236

v086

claude/improve-test-suite-xm6Bh

claude/typescript-archivebox-rewrite-011CUmPyEmECnkXwyHRT9RPF

claude/archivebox-schema-orm-comparison-011CUn26GurEpLiBUq4zjVLJ

stable

logfire

plugins-browsertrix

v072

link-removal2

v0.8.6rc1

v0.7.3

v0.8.6rc0

v0.8.5rc53

v0.8.5rc51

v0.8.5rc50

v0.8.5rc49

v0.8.5rc48

v0.8.5rc47

v0.8.5rc46

v0.8.5rc45

v0.8.5rc44

v0.8.5rc43

v0.8.5rc42

v0.8.5rc41

v0.8.5rc40

v0.8.5rc39

v0.8.5rc38

v0.8.5rc37

v0.8.5rc36

v0.8.5rc35

v0.8.5rc34

v0.8.5rc33

v0.8.5rc32

v0.8.5rc31

v0.8.5rc30

v0.8.5rc29

v0.8.5rc28

v0.8.5rc27

v0.8.5rc26

v0.8.5rc25

v0.8.5rc24

v0.8.5rc23

v0.8.5rc22

v0.8.5rc13

v0.8.5rc12

v0.8.5rc11

v0.8.5rc10

v0.8.5rc9

v0.8.5rc8

v0.8.5rc6

v0.8.5rc5

v0.8.5rc4

v0.8.5rc3

v0.8.5rc2

v0.8.5-rc

v0.8.4-rc

v0.8.3-rc

v0.8.2-rc

v0.8.0-rc

v0.7.2

v0.7.1

v0.7.0

v0.6.2

v0.6.0

v0.5.6

v0.5.4

v0.5.3

v0.4.24

v0.4.21

v0.4.20

v0.4.19

v0.4.18

v0.4.17

v0.4.16

v0.4.15

v0.4.14

v0.4.13

v0.4.12

v0.4.11

v0.4.9

v0.2.4

v0.2.3

v0.2.2

v0.2.1

v0.2.0

v0.1.0

v0.0.3

v0.0.2

v0.0.1

Labels

Clear labels

expected: maybe someday

expected: next release

expected: release after next

expected: unlikely unless contributed

good first ticket

help wanted

pull-request

Mirrored from GitHub Pull Request

status: needs followup

status: wip

status: wontfix

touches: API/CLI/Spec

touches: configuration

touches: data/schema/architecture

touches: dependencies/packaging

touches: docs

touches: js

touches: views/replayers/html/css

No labels

expected: maybe someday

expected: next release

expected: release after next

expected: unlikely unless contributed

status: needs followup

status: wip

status: wontfix

touches: API/CLI/Spec

touches: configuration

touches: data/schema/architecture

touches: dependencies/packaging

touches: docs

touches: js

touches: views/replayers/html/css

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#1161

Reference in a new issue

Repository

starred/ArchiveBox

Title

Body

No description provided.

Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?

Rows
Columns