starred/ArchiveBox
1
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 17:16:00 +03:00
Code Issues 624 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1052] Question: Can't set up ArchiveBox DB on a virtio mount that doesn't support FSYNC (sqlite3.OperationalError: disk I/O error) #659

New issue
Closed
opened 2026-03-01 14:45:21 +03:00 by kerem · 1 comment
kerem commented 2026-03-01 14:45:21 +03:00
Owner
Copy link

Originally created by @rebane2001 on GitHub (Nov 22, 2022).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1052

Describe the bug

Trying to set up ArchiveBox on a virtio mount results in an error (sqlite3.OperationalError: disk I/O error).

Screenshots or log output

fstab of the affected mount:

archive        /mnt/archive            9p         trans=virtio,version=9p2000.L,_netdev,rw 0 0

archivebox init logs:

$ docker compose run archivebox init --setup
[i] [2022-11-22 13:05:09] ArchiveBox v0.6.2: archivebox init --setup
    > /data

[+] Initializing a new ArchiveBox v0.6.2 collection...
----------------------------------------------------------------------

[+] Building archive folder structure...
    + ./archive, ./sources, ./logs...
    + ./ArchiveBox.conf...

[+] Building main SQL index and running initial migrations...
    Operations to perform:
      Apply all migrations: admin, auth, contenttypes, core, sessions
    Running migrations:
    Applying contenttypes.0001_initial... OK
    Applying auth.0001_initial... OK
    Applying admin.0001_initial... OK
    Applying admin.0002_logentry_remove_auto_add... OK
    Applying admin.0003_logentry_add_action_flag_choices... OK
    Applying contenttypes.0002_remove_content_type_name... OK
    Applying auth.0002_alter_permission_name_max_length... OK
    Applying auth.0003_alter_user_email_max_length... OK
    Applying auth.0004_alter_user_username_opts... OK
    Applying auth.0005_alter_user_last_login_null... OK
    Applying auth.0006_require_contenttypes_0002... OK
    Applying auth.0007_alter_validators_add_error_messages... OK
    Applying auth.0008_alter_user_username_max_length... OK
    Applying auth.0009_alter_user_last_name_max_length... OK
    Applying auth.0010_alter_group_name_max_length... OK
    Applying auth.0011_update_proxy_permissions... OK
    Applying auth.0012_alter_user_first_name_max_length... OK
    Applying core.0001_initial... OK
    Applying core.0002_auto_20200625_1521... OK
    Applying core.0003_auto_20200630_1034... OK
    Applying core.0004_auto_20200713_1552... OK
    Applying core.0005_auto_20200728_0326... OK
    Applying core.0006_auto_20201012_1520... OK
    Applying core.0007_archiveresult... OK
    Applying core.0008_auto_20210105_1421... OK
    Applying core.0009_auto_20210216_1038... OK
    Applying core.0010_auto_20210216_1055... OK
    Applying core.0011_auto_20210216_1331... OK
    Applying core.0012_auto_20210216_1425... OK
    Applying core.0013_auto_20210218_0729... OK
    Applying core.0014_auto_20210218_0729... OK
    Applying core.0015_auto_20210218_0730... OK
    Applying core.0016_auto_20210218_1204... OK
    Applying core.0017_auto_20210219_0211... OK
    Applying core.0018_auto_20210327_0952... OK
    Applying core.0019_auto_20210401_0654... OK
    Applying core.0020_auto_20210410_1031... OK
    Applying sessions.0001_initial... OK

    √ ./index.sqlite3

[*] Checking links from indexes and archive folders (safe to Ctrl+C)...

[*] [2022-11-22 13:05:36] Writing 0 links to main index...
    √ ./index.sqlite3

----------------------------------------------------------------------
[√] Done. A new ArchiveBox collection was initialized (0 links).
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')())
  File "/app/archivebox/cli/__init__.py", line 140, in main
    run_subcommand(
  File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/app/archivebox/cli/archivebox_init.py", line 43, in main
    init(
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/main.py", line 420, in init
    run_subcommand('setup', pwd=out_dir)
  File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/app/archivebox/cli/archivebox_setup.py", line 33, in main
    setup(
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/main.py", line 906, in setup
    if not User.objects.filter(is_superuser=True).exists():
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 809, in exists
    return self.query.has_results(using=self.db)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 535, in has_results
    return compiler.has_results()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: disk I/O error

ArchiveBox version

ArchiveBox v0.6.2
Cpython Linux Linux-5.15.0-52-generic-x86_64-with-glibc2.28 x86_64
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.6.2          valid     /usr/local/bin/archivebox                                          
 √  PYTHON_BINARY         v3.9.5          valid     /usr/local/bin/python3.9                                           
 √  DJANGO_BINARY         v3.1.10         valid     /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py  
 √  CURL_BINARY           v7.64.0         valid     /usr/bin/curl                                                      
 √  WGET_BINARY           v1.20.1         valid     /usr/bin/wget                                                      
 √  NODE_BINARY           v15.14.0        valid     /usr/bin/node                                                      
 √  SINGLEFILE_BINARY     v0.3.16         valid     /node/node_modules/single-file/cli/single-file                     
 √  READABILITY_BINARY    v0.0.2          valid     /node/node_modules/readability-extractor/readability-extractor     
 √  MERCURY_BINARY        v1.0.0          valid     /node/node_modules/@postlight/mercury-parser/cli.js                
 √  GIT_BINARY            v2.20.1         valid     /usr/bin/git                                                       
 √  YOUTUBEDL_BINARY      v2021.04.26     valid     /usr/local/bin/youtube-dl                                          
 √  CHROME_BINARY         v90.0.4430.93   valid     /usr/bin/chromium                                                  
 √  RIPGREP_BINARY        v0.10.0         valid     /usr/bin/rg                                                        

[i] Source-code locations:
 √  PACKAGE_DIR           22 files        valid     /app/archivebox                                                    
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                          
 -  CUSTOM_TEMPLATES_DIR  -               disabled                                                                     

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                     
 -  COOKIES_FILE          -               disabled                                                                     


[i] Data locations:
Originally created by @rebane2001 on GitHub (Nov 22, 2022). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1052 #### Describe the bug Trying to set up ArchiveBox on a virtio mount results in an error (`sqlite3.OperationalError: disk I/O error`). #### Screenshots or log output fstab of the affected mount: ``` archive /mnt/archive 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0 ``` archivebox init logs: ```logs $ docker compose run archivebox init --setup [i] [2022-11-22 13:05:09] ArchiveBox v0.6.2: archivebox init --setup > /data [+] Initializing a new ArchiveBox v0.6.2 collection... ---------------------------------------------------------------------- [+] Building archive folder structure... + ./archive, ./sources, ./logs... + ./ArchiveBox.conf... [+] Building main SQL index and running initial migrations... Operations to perform: Apply all migrations: admin, auth, contenttypes, core, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying core.0001_initial... OK Applying core.0002_auto_20200625_1521... OK Applying core.0003_auto_20200630_1034... OK Applying core.0004_auto_20200713_1552... OK Applying core.0005_auto_20200728_0326... OK Applying core.0006_auto_20201012_1520... OK Applying core.0007_archiveresult... OK Applying core.0008_auto_20210105_1421... OK Applying core.0009_auto_20210216_1038... OK Applying core.0010_auto_20210216_1055... OK Applying core.0011_auto_20210216_1331... OK Applying core.0012_auto_20210216_1425... OK Applying core.0013_auto_20210218_0729... OK Applying core.0014_auto_20210218_0729... OK Applying core.0015_auto_20210218_0730... OK Applying core.0016_auto_20210218_1204... OK Applying core.0017_auto_20210219_0211... OK Applying core.0018_auto_20210327_0952... OK Applying core.0019_auto_20210401_0654... OK Applying core.0020_auto_20210410_1031... OK Applying sessions.0001_initial... OK √ ./index.sqlite3 [*] Checking links from indexes and archive folders (safe to Ctrl+C)... [*] [2022-11-22 13:05:36] Writing 0 links to main index... √ ./index.sqlite3 ---------------------------------------------------------------------- [√] Done. A new ArchiveBox collection was initialized (0 links). Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute return Database.Cursor.execute(self, query, params) sqlite3.OperationalError: disk I/O error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/bin/archivebox", line 33, in <module> sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')()) File "/app/archivebox/cli/__init__.py", line 140, in main run_subcommand( File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore File "/app/archivebox/cli/archivebox_init.py", line 43, in main init( File "/app/archivebox/util.py", line 114, in typechecked_function return func(*args, **kwargs) File "/app/archivebox/main.py", line 420, in init run_subcommand('setup', pwd=out_dir) File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore File "/app/archivebox/cli/archivebox_setup.py", line 33, in main setup( File "/app/archivebox/util.py", line 114, in typechecked_function return func(*args, **kwargs) File "/app/archivebox/main.py", line 906, in setup if not User.objects.filter(is_superuser=True).exists(): File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 809, in exists return self.query.has_results(using=self.db) File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 535, in has_results return compiler.has_results() File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results return bool(self.execute_sql(SINGLE)) File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute return Database.Cursor.execute(self, query, params) django.db.utils.OperationalError: disk I/O error ``` #### ArchiveBox version <!-- Run the `archivebox version` command locally then copy paste the result here: --> ```logs ArchiveBox v0.6.2 Cpython Linux Linux-5.15.0-52-generic-x86_64-with-glibc2.28 x86_64 IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep [i] Dependency versions: √ ARCHIVEBOX_BINARY v0.6.2 valid /usr/local/bin/archivebox √ PYTHON_BINARY v3.9.5 valid /usr/local/bin/python3.9 √ DJANGO_BINARY v3.1.10 valid /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py √ CURL_BINARY v7.64.0 valid /usr/bin/curl √ WGET_BINARY v1.20.1 valid /usr/bin/wget √ NODE_BINARY v15.14.0 valid /usr/bin/node √ SINGLEFILE_BINARY v0.3.16 valid /node/node_modules/single-file/cli/single-file √ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor √ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js √ GIT_BINARY v2.20.1 valid /usr/bin/git √ YOUTUBEDL_BINARY v2021.04.26 valid /usr/local/bin/youtube-dl √ CHROME_BINARY v90.0.4430.93 valid /usr/bin/chromium √ RIPGREP_BINARY v0.10.0 valid /usr/bin/rg [i] Source-code locations: √ PACKAGE_DIR 22 files valid /app/archivebox √ TEMPLATES_DIR 3 files valid /app/archivebox/templates - CUSTOM_TEMPLATES_DIR - disabled [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled - COOKIES_FILE - disabled [i] Data locations: ``` <!-- Tickets without full version info will closed until it is provided, we need the full output here to help you solve your issue -->
kerem 2026-03-01 14:45:21 +03:00
kerem commented 2026-03-01 14:45:21 +03:00
Author
Owner
Copy link

@pirate commented on GitHub (Jun 13, 2023):

Sorry for the late response, unfortunately we don't support saving the SQLite DB on a filesystem that doesn't support FSYNC at the moment (as it can cause data loss when running multiple ArchiveBox commands at once).
You can find more information here: https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#filesystem

There is an undocumented footgun provided if you're willing to accept possible data loss when running multiple archivebox instances in parallel: set archivebox config ENFORCE_ATOMIC_WRITES=False https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config.py#L87.

Comment back here if you're still having problems and I can re-open the issue :)

<!-- gh-comment-id:1589085036 --> @pirate commented on GitHub (Jun 13, 2023): Sorry for the late response, unfortunately we don't support saving the SQLite DB on a filesystem that doesn't support FSYNC at the moment (as it can cause data loss when running multiple ArchiveBox commands at once). You can find more information here: https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#filesystem There is an undocumented footgun provided if you're willing to accept possible data loss when running multiple archivebox instances in parallel: set `archivebox config ENFORCE_ATOMIC_WRITES=False` https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config.py#L87. Comment back here if you're still having problems and I can re-open the issue :)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
dev
main
abx-dl-refactor
claude/issue-1688-20251229-2233
claude/s3-support-migration-JZQcf
claude/review-snapshot-archive-RJOkr
claude/review-code-quality-macdO
claude/reduce-chrome-duplication-JgWVr
fix/chrome-args-comma-parsing
claude/issue-668-20251229-2145
claude/issue-1664-20251229-2236
v086
claude/improve-test-suite-xm6Bh
claude/typescript-archivebox-rewrite-011CUmPyEmECnkXwyHRT9RPF
claude/archivebox-schema-orm-comparison-011CUn26GurEpLiBUq4zjVLJ
stable
logfire
plugins-browsertrix
v072
link-removal2
v0.8.6rc1
v0.7.3
v0.8.6rc0
v0.8.5rc53
v0.8.5rc51
v0.8.5rc50
v0.8.5rc49
v0.8.5rc48
v0.8.5rc47
v0.8.5rc46
v0.8.5rc45
v0.8.5rc44
v0.8.5rc43
v0.8.5rc42
v0.8.5rc41
v0.8.5rc40
v0.8.5rc39
v0.8.5rc38
v0.8.5rc37
v0.8.5rc36
v0.8.5rc35
v0.8.5rc34
v0.8.5rc33
v0.8.5rc32
v0.8.5rc31
v0.8.5rc30
v0.8.5rc29
v0.8.5rc28
v0.8.5rc27
v0.8.5rc26
v0.8.5rc25
v0.8.5rc24
v0.8.5rc23
v0.8.5rc22
v0.8.5rc13
v0.8.5rc12
v0.8.5rc11
v0.8.5rc10
v0.8.5rc9
v0.8.5rc8
v0.8.5rc6
v0.8.5rc5
v0.8.5rc4
v0.8.5rc3
v0.8.5rc2
v0.8.5-rc
v0.8.4-rc
v0.8.3-rc
v0.8.2-rc
v0.8.0-rc
v0.7.2
v0.7.1
v0.7.0
v0.6.2
v0.6.0
v0.5.6
v0.5.4
v0.5.3
v0.4.24
v0.4.21
v0.4.20
v0.4.19
v0.4.18
v0.4.17
v0.4.16
v0.4.15
v0.4.14
v0.4.13
v0.4.12
v0.4.11
v0.4.9
v0.2.4
v0.2.3
v0.2.2
v0.2.1
v0.2.0
v0.1.0
v0.0.3
v0.0.2
v0.0.1
Labels
Clear labels   
expected: maybe someday

   
expected: next release

   
expected: release after next

   
expected: unlikely unless contributed

   
good first ticket

   
help wanted

   
pull-request

Mirrored from GitHub Pull Request

  
scope: all users

   
scope: windows users

   
size: easy

   
size: hard

   
size: medium

   
size: medium

   
status: backlog

   
status: blocked

   
status: done

   
status: idea-phase

   
status: needs followup

   
status: wip

   
status: wontfix

   
touches: API/CLI/Spec

   
touches: configuration

   
touches: data/schema/architecture

   
touches: dependencies/packaging

   
touches: docs

   
touches: js

   
touches: views/replayers/html/css

   
why: correctness

   
why: functionality

   
why: performance

   
why: security
No labels
expected: maybe someday
expected: next release
expected: release after next
expected: unlikely unless contributed
good first ticket
help wanted
pull-request
scope: all users
scope: windows users
size: easy
size: hard
size: medium
size: medium
status: backlog
status: blocked
status: done
status: idea-phase
status: needs followup
status: wip
status: wontfix
touches: API/CLI/Spec
touches: configuration
touches: data/schema/architecture
touches: dependencies/packaging
touches: docs
touches: js
touches: views/replayers/html/css
why: correctness
why: functionality
why: performance
why: security
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/ArchiveBox#659
No description provided.