starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 09:06:02 +03:00

[GH-ISSUE #1005] Feature Request: How to change db engine to mysql or mangodb #628

New issue

Closed

opened 2026-03-01 14:45:09 +03:00 by kerem · 3 comments

kerem commented

2026-03-01 14:45:09 +03:00

Owner

Originally created by @thinker007 on GitHub (Jul 27, 2022).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1005

Type

[ x] General question or discussion
[x ] Request modification of existing behavior or design

What is the problem that your feature request solves

I need to archive large scale web page, need a better db solution.

Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes

change db from sqlite3 to mysql or mangodb for better archive performance and large scale web page archieve

What hacks or alternative solutions have you tried to solve the problem?

move data from sqlite3 to mysql manually.

How badly do you want this new feature?

[x ] It's an urgent deal-breaker, I can't live without it

I'm willing to contribute dev time / money to fix this issue
I like ArchiveBox so far / would recommend it to a friend
I've had a lot of difficulty getting ArchiveBox set up

Originally created by @thinker007 on GitHub (Jul 27, 2022). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1005  ## Type - [ x] General question or discussion - [x ] Request modification of existing behavior or design ## What is the problem that your feature request solves I need to archive large scale web page, need a better db solution. ## Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes change db from sqlite3 to mysql or mangodb for better archive performance and large scale web page archieve ## What hacks or alternative solutions have you tried to solve the problem? move data from sqlite3 to mysql manually. ## How badly do you want this new feature? - [x ] It's an urgent deal-breaker, I can't live without it --- - [x] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue - [x] I like ArchiveBox so far / would recommend it to a friend - [ ] I've had a lot of difficulty getting ArchiveBox set up

kerem

2026-03-01 14:45:09 +03:00

closed this issue
added the
status: idea-phase
label

kerem commented

2026-03-01 14:45:10 +03:00

Author

Owner

@pirate commented on GitHub (Jul 30, 2022):

Not supported currently and not going to support this within the next few years (or likely ever), sorry. SQLite is a much more durable data format than Mysql or Postgresql, and it's easier on many levels because of the lack of need to run a dedicated server process.

large scale web page, need a better db solution.

This is also a common misconception. SQLite can handle multi terabyte DBs and massive scale applications, it's more about the need for concurrent writers than the size of the DB or anything "web scale".

See here for more explanation:

@pirate commented on GitHub (Jul 30, 2022): Not supported currently and not going to support this within the next few years (or likely ever), sorry. SQLite is a much more durable data format than Mysql or Postgresql, and it's easier on many levels because of the lack of need to run a dedicated server process. > large scale web page, need a better db solution. This is also a common misconception. SQLite can handle multi terabyte DBs and massive scale applications, it's more about the need for concurrent writers than the size of the DB or anything "web scale". See here for more explanation: - https://github.com/ArchiveBox/ArchiveBox/issues/601#issuecomment-817446169 - https://github.com/ArchiveBox/ArchiveBox/issues/787

kerem commented

2026-03-01 14:45:10 +03:00

Author

Owner

@thinker007 commented on GitHub (Aug 17, 2022):

when I try to archive 1 million urls, then the db became a bottle neck ! the db lock problem pops up frequently

@thinker007 commented on GitHub (Aug 17, 2022): when I try to archive 1 million urls, then the db became a bottle neck ! the db lock problem pops up frequently

kerem commented

2026-03-01 14:45:10 +03:00

Author

Owner

@pirate commented on GitHub (Aug 20, 2022):

Are you archiving with multiple parallel processes? With a single process there is no db lock contention. It's more likely the parallel processes that are causing lock contention, not the size of the archive.

Please screenshot the errors you see with any log output.

@pirate commented on GitHub (Aug 20, 2022): Are you archiving with multiple parallel processes? With a single process there is no db lock contention. It's more likely the parallel processes that are causing lock contention, not the size of the archive. Please screenshot the errors you see with any log output.

kerem referenced this issue

2026-03-01 14:49:02 +03:00

[PR #628] [MERGED] fix: Refactor should_save_extractor methods to accept overwrite parameter #1253

kerem referenced this issue

2026-03-01 18:00:40 +03:00

[PR #628] fix: Refactor should_save_extractor methods to accept overwrite parameter #2762

kerem referenced this issue

2026-03-15 01:35:16 +03:00

[PR #628] [MERGED] fix: Refactor should_save_extractor methods to accept overwrite parameter #4265