mirror of
https://github.com/ArchiveBox/ArchiveBox.git
synced 2026-04-25 17:16:00 +03:00
[GH-ISSUE #57] New index single-source-of-truth instead of JSON: SQL Database w/ migrations #1552
Labels
No labels
expected: maybe someday
expected: next release
expected: release after next
expected: unlikely unless contributed
good first ticket
help wanted
pull-request
scope: all users
scope: windows users
size: easy
size: hard
size: medium
size: medium
status: backlog
status: blocked
status: done
status: idea-phase
status: needs followup
status: wip
status: wontfix
touches: API/CLI/Spec
touches: configuration
touches: data/schema/architecture
touches: dependencies/packaging
touches: docs
touches: js
touches: views/replayers/html/css
why: correctness
why: functionality
why: performance
why: security
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/ArchiveBox#1552
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @pirate on GitHub (Dec 20, 2017).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/57
Originally assigned to: @pirate on GitHub.
If I end up writing a backend, Django is not a bad choice because the migrations system would allow us to keep an up-to-date queryable SQLite database with all the links.
Running the archive.py script at any point would update the schema to the latest version, because it has the entire migration history from the time the index was last exported till now.
On every run, the json can be re-exported and the files updated if necessary. The SQLite database would also allow for fast searching and indexing of the links.
Files can be foreign-keyed on both uuid and their sha-512 hash to globally deduplicate and enable sharding later.
@pirate commented on GitHub (Feb 23, 2018):
Upcoming design: https://docs.google.com/presentation/d/17gPkY6chAzDDXCce1QkTdRwl2nChESdjfibal-n8TgM/edit?usp=sharing
@FiloSottile commented on GitHub (Dec 1, 2018):
SQLite also come with nice full-text search (FTE5) which works even if you don't store the actual content in it, so you can have all the outputs in files, and use SQLite as the index and search solution. (Would go well with #69.)
@pirate commented on GitHub (Dec 1, 2018):
I experimented with ElasticLunr earlier but SQLite's FTE5 looks like a great alternative. If we use Django it'll be db-agnostic and we can do full-text search on both SQLite and Postgres.
@cdvv7788 commented on GitHub (Jul 16, 2020):
djangobranch has progress in this regard. sqlite database is already working along the legacy indexes.@pirate commented on GitHub (Jul 16, 2020):
Closing this because as @cdvv7788 said, the
djangowith the next release has a full sqlite3 db with migrations. We can work on making it the single-source-of-truth gradually over several subsequent versions. Until then it will use all three indexes together.v0.7.1on alpine linux OSError: crontab: can't open 'archivebox': No such file or directory #2432v0.7.1on alpine linux OSError: crontab: can't open 'archivebox': No such file or directory #3943