starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 17:16:00 +03:00

[GH-ISSUE #57] New index single-source-of-truth instead of JSON: SQL Database w/ migrations #3060

New issue

Closed

opened 2026-03-14 20:50:24 +03:00 by kerem · 5 comments

kerem commented

2026-03-14 20:50:24 +03:00

Owner

Originally created by @pirate on GitHub (Dec 20, 2017).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/57

Originally assigned to: @pirate on GitHub.

If I end up writing a backend, Django is not a bad choice because the migrations system would allow us to keep an up-to-date queryable SQLite database with all the links.

Running the archive.py script at any point would update the schema to the latest version, because it has the entire migration history from the time the index was last exported till now.

On every run, the json can be re-exported and the files updated if necessary. The SQLite database would also allow for fast searching and indexing of the links.

Files can be foreign-keyed on both uuid and their sha-512 hash to globally deduplicate and enable sharding later.

Originally created by @pirate on GitHub (Dec 20, 2017). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/57 Originally assigned to: @pirate on GitHub. If I end up writing a backend, Django is not a bad choice because the migrations system would allow us to keep an up-to-date queryable SQLite database with all the links. Running the archive.py script at any point would update the schema to the latest version, because it has the entire migration history from the time the index was last exported till now. On every run, the json can be re-exported and the files updated if necessary. The SQLite database would also allow for fast searching and indexing of the links. Files can be foreign-keyed on both uuid and their sha-512 hash to globally deduplicate and enable sharding later.

kerem

2026-03-14 20:50:24 +03:00

closed this issue
added the
size: hard

status: idea-phase
labels

kerem commented

2026-03-14 20:50:41 +03:00

Author

Owner

@pirate commented on GitHub (Feb 23, 2018):

Upcoming design: https://docs.google.com/presentation/d/17gPkY6chAzDDXCce1QkTdRwl2nChESdjfibal-n8TgM/edit?usp=sharing

@pirate commented on GitHub (Feb 23, 2018): Upcoming design: https://docs.google.com/presentation/d/17gPkY6chAzDDXCce1QkTdRwl2nChESdjfibal-n8TgM/edit?usp=sharing

kerem commented

2026-03-14 20:50:46 +03:00

Author

Owner

@FiloSottile commented on GitHub (Dec 1, 2018):

SQLite also come with nice full-text search (FTE5) which works even if you don't store the actual content in it, so you can have all the outputs in files, and use SQLite as the index and search solution. (Would go well with #69.)

@FiloSottile commented on GitHub (Dec 1, 2018): SQLite also come with nice full-text search (FTE5) which works even if you don't store the actual content in it, so you can have all the outputs in files, and use SQLite as the index and search solution. (Would go well with #69.)

kerem commented

2026-03-14 20:50:51 +03:00

Author

Owner

@pirate commented on GitHub (Dec 1, 2018):

I experimented with ElasticLunr earlier but SQLite's FTE5 looks like a great alternative. If we use Django it'll be db-agnostic and we can do full-text search on both SQLite and Postgres.

@pirate commented on GitHub (Dec 1, 2018): I experimented with ElasticLunr earlier but SQLite's FTE5 looks like a great alternative. If we use Django it'll be db-agnostic and we can do full-text search on both SQLite and Postgres.

kerem commented

2026-03-14 20:50:56 +03:00

Author

Owner

@cdvv7788 commented on GitHub (Jul 16, 2020):

django branch has progress in this regard. sqlite database is already working along the legacy indexes.

@cdvv7788 commented on GitHub (Jul 16, 2020): `django` branch has progress in this regard. sqlite database is already working along the legacy indexes.

kerem commented

2026-03-14 20:51:01 +03:00

Author

Owner

@pirate commented on GitHub (Jul 16, 2020):

Closing this because as @cdvv7788 said, the django with the next release has a full sqlite3 db with migrations. We can work on making it the single-source-of-truth gradually over several subsequent versions. Until then it will use all three indexes together.

@pirate commented on GitHub (Jul 16, 2020): Closing this because as @cdvv7788 said, the `django` with the next release has a full sqlite3 db with migrations. We can work on making it the single-source-of-truth gradually over several subsequent versions. Until then it will use all three indexes together.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#3060

No description provided.

Rows
Columns