starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 09:06:02 +03:00

[GH-ISSUE #1178] Bug: Doing a later, successful "Pull" on snapshots classified as "failed" doesn't change their status away from "failed" #2242

New issue

Open

opened 2026-03-01 17:57:36 +03:00 by kerem · 5 comments

kerem commented

2026-03-01 17:57:36 +03:00

Owner

Originally created by @melyux on GitHub (Jul 13, 2023).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1178

Describe the bug

If one of the extractors for a snapshot fails to get its content, the snapshot is classified as "Failed". If you later go in and do a "Pull" on it with the button, it retries these failed extractors. If this operation succeeds, the snapshot's status does not get moved from "failed" to "succeeded".

The "status" in the filter seems to apply to the individual extractor results inside snapshots rather than the snapshots themselves, since this snapshot shows up under both "succeeded" and "failed", which is weird. Once everything works upon subsequent "Pull"s, the error count and the failed statuses should be removed, I think. Otherwise there's no point to these filters

Steps to reproduce

Add a URL and have one or more of its extractors fail.
Run a "Pull" on the snapshot, and this time have the extractors succeed.
See that the snapshot still shows up when filtering by status "failed".

Screenshots or log output

ArchiveBox version

find: '/.config/chromium/Crash Reports/pending/': No such file or directory
0.6.3
ArchiveBox v0.6.3 Cpython Linux Linux-6.1.0-10-amd64-x86_64-with-glibc2.31 x86_64
DEBUG=False IN_DOCKER=True IS_TTY=True TZ=UTC FS_ATOMIC=True FS_REMOTE=True FS_PERMS=644 999:999 SEARCH_BACKEND=ripgrep

[i] Dependency versions:
 √  PYTHON_BINARY         v3.11.3         valid     /usr/local/bin/python3.11                                                   
 √  SQLITE_BINARY         v2.6.0          valid     /usr/local/lib/python3.11/sqlite3/dbapi2.py                                 
 √  DJANGO_BINARY         v3.1.14         valid     /usr/local/lib/python3.11/site-packages/django/__init__.py                  
 √  ARCHIVEBOX_BINARY     v0.6.3          valid     /usr/local/bin/archivebox                                                   

 √  CURL_BINARY           v7.74.0         valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.21           valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v18.16.0        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v0.3.16         valid     /node/node_modules/single-file/cli/single-file                              
 √  READABILITY_BINARY    v0.0.2          valid     /node/node_modules/readability-extractor/readability-extractor              
 √  MERCURY_BINARY        v1.0.0          valid     /node/node_modules/@postlight/mercury-parser/cli.js                         
 √  GIT_BINARY            v2.30.2         valid     /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2023.03.04     valid     /usr/local/bin/yt-dlp                                                       
 √  CHROME_BINARY         v112.0.5615.138  valid     /usr/bin/chromium                                                           
 √  RIPGREP_BINARY        v12.1.1         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           22 files        valid     /app/archivebox                                                             
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                                   
 -  CUSTOM_TEMPLATES_DIR  -               disabled                                                                              

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                              
 -  COOKIES_FILE          -               disabled                                                                              

[i] Data locations:
 √  OUTPUT_DIR            7 files @       valid     /data                                                                       
 √  SOURCES_DIR           172 files       valid     ./sources                                                                   
 √  LOGS_DIR              2 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           54 files        valid     ./archive                                                                   
 √  CONFIG_FILE           81.0 Bytes      valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             776.0 KB        valid     ./index.sqlite3

Originally created by @melyux on GitHub (Jul 13, 2023). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1178  #### Describe the bug  If one of the extractors for a snapshot fails to get its content, the snapshot is classified as "Failed". If you later go in and do a "Pull" on it with the button, it retries these failed extractors. If this operation succeeds, the snapshot's status does not get moved from "failed" to "succeeded". The "status" in the filter seems to apply to the individual extractor results inside snapshots rather than the snapshots themselves, since this snapshot shows up under both "succeeded" and "failed", which is weird. Once everything works upon subsequent "Pull"s, the error count and the failed statuses should be removed, I think. Otherwise there's no point to these filters #### Steps to reproduce  1. Add a URL and have one or more of its extractors fail. 2. Run a "Pull" on the snapshot, and this time have the extractors succeed. 3. See that the snapshot still shows up when filtering by status "failed". #### Screenshots or log output  #### ArchiveBox version  ```logs find: '/.config/chromium/Crash Reports/pending/': No such file or directory 0.6.3 ArchiveBox v0.6.3 Cpython Linux Linux-6.1.0-10-amd64-x86_64-with-glibc2.31 x86_64 DEBUG=False IN_DOCKER=True IS_TTY=True TZ=UTC FS_ATOMIC=True FS_REMOTE=True FS_PERMS=644 999:999 SEARCH_BACKEND=ripgrep [i] Dependency versions: √ PYTHON_BINARY v3.11.3 valid /usr/local/bin/python3.11 √ SQLITE_BINARY v2.6.0 valid /usr/local/lib/python3.11/sqlite3/dbapi2.py √ DJANGO_BINARY v3.1.14 valid /usr/local/lib/python3.11/site-packages/django/__init__.py √ ARCHIVEBOX_BINARY v0.6.3 valid /usr/local/bin/archivebox √ CURL_BINARY v7.74.0 valid /usr/bin/curl √ WGET_BINARY v1.21 valid /usr/bin/wget √ NODE_BINARY v18.16.0 valid /usr/bin/node √ SINGLEFILE_BINARY v0.3.16 valid /node/node_modules/single-file/cli/single-file √ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor √ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js √ GIT_BINARY v2.30.2 valid /usr/bin/git √ YOUTUBEDL_BINARY v2023.03.04 valid /usr/local/bin/yt-dlp √ CHROME_BINARY v112.0.5615.138 valid /usr/bin/chromium √ RIPGREP_BINARY v12.1.1 valid /usr/bin/rg [i] Source-code locations: √ PACKAGE_DIR 22 files valid /app/archivebox √ TEMPLATES_DIR 3 files valid /app/archivebox/templates - CUSTOM_TEMPLATES_DIR - disabled [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled - COOKIES_FILE - disabled [i] Data locations: √ OUTPUT_DIR 7 files @ valid /data √ SOURCES_DIR 172 files valid ./sources √ LOGS_DIR 2 files valid ./logs √ ARCHIVE_DIR 54 files valid ./archive √ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf √ SQL_INDEX 776.0 KB valid ./index.sqlite3 ```

kerem added the

touches: data/schema/architecture

size: medium

expected: next release

status: backlog

scope: all users

why: correctness

labels

2026-03-01 17:57:36 +03:00

kerem commented

2026-03-01 17:57:37 +03:00

Author

Owner

@melyux commented on GitHub (Jul 15, 2023):

Thought about this for a while, and snapshot filters should work like this: if the latest update was successful for an extractor for that snapshot, previous failures of that extractor should not count against the snapshot as "failed". Only when the latest update for at least one extractor failed should that snapshot be designated as "failed".

@melyux commented on GitHub (Jul 15, 2023): Thought about this for a while, and snapshot filters should work like this: if the latest update was successful for an extractor for that snapshot, previous failures of that extractor should not count against the snapshot as "failed". Only when the latest update for at least one extractor failed should that snapshot be designated as "failed".

kerem commented

2026-03-01 17:57:37 +03:00

Author

Owner

@pirate commented on GitHub (Aug 13, 2023):

Yeah I agree @melyux that's how they were intended to work already, there must be a bug in the filter logic.

@pirate commented on GitHub (Aug 13, 2023): Yeah I agree @melyux that's how they were intended to work already, there must be a bug in the filter logic.

kerem commented

2026-03-01 17:57:37 +03:00

Author

Owner

@neel-suthar commented on GitHub (Jan 22, 2024):

I thought Snapshots had some kind of status field but seems like I am wrong. But is it worth it to add a status field for each snapshot? Internally the logic should stay the same but this can help us fetch snapshots very effectively. Just a thought.

@neel-suthar commented on GitHub (Jan 22, 2024): I thought Snapshots had some kind of status field but seems like I am wrong. But is it worth it to add a status field for each snapshot? Internally the logic should stay the same but this can help us fetch snapshots very effectively. Just a thought.

kerem commented

2026-03-01 17:57:37 +03:00

Author

Owner

@pirate commented on GitHub (Jan 23, 2024):

Nah @neel-suthar, I want Snapshots to stay basically immutable (i.e. no flag/status/etc fields) because we're moving to an event-driven model soon. But we can add a @cached_property that gets the status using a query over ArchiveResults and stores it in cache.

@pirate commented on GitHub (Jan 23, 2024): Nah @neel-suthar, I want Snapshots to stay basically immutable (i.e. no flag/status/etc fields) because we're moving to an event-driven model soon. But we can add a `@cached_property` that gets the status using a query over `ArchiveResult`s and stores it in cache.

kerem commented

2026-03-01 17:57:37 +03:00

Author

Owner

@pirate commented on GitHub (Oct 27, 2024):

This longstanding bug should soon be fixed, each model is now a finite state machine with only a few valid states. Everything gets moved towards a final state deterministically on tick() (like a game engine), and if a snapshot fails enough times it will eventually be marked "fatal", and will have to be retried as a new snapshot. This should make it much clearer when something is failing intermittently vs permanently.

@pirate commented on GitHub (Oct 27, 2024): This longstanding bug should soon be fixed, each model is now a finite state machine with only a few valid states. Everything gets moved towards a final state deterministically on `tick()` (like a game engine), and if a snapshot fails enough times it will eventually be marked "fatal", and will have to be retried as a new snapshot. This should make it much clearer when something is failing intermittently vs permanently.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#2242

No description provided.

Rows
Columns