starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 17:16:00 +03:00

[GH-ISSUE #1052] Question: Can't set up ArchiveBox DB on a virtio mount that doesn't support FSYNC `(sqlite3.OperationalError: disk I/O error)` #3680

New issue

Closed

opened 2026-03-15 00:00:17 +03:00 by kerem · 1 comment

kerem commented

2026-03-15 00:00:17 +03:00

Owner

Originally created by @rebane2001 on GitHub (Nov 22, 2022).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1052

Describe the bug

Trying to set up ArchiveBox on a virtio mount results in an error (sqlite3.OperationalError: disk I/O error).

Screenshots or log output

fstab of the affected mount:

archive        /mnt/archive            9p         trans=virtio,version=9p2000.L,_netdev,rw 0 0

archivebox init logs:

$ docker compose run archivebox init --setup
[i] [2022-11-22 13:05:09] ArchiveBox v0.6.2: archivebox init --setup
    > /data

[+] Initializing a new ArchiveBox v0.6.2 collection...
----------------------------------------------------------------------

[+] Building archive folder structure...
    + ./archive, ./sources, ./logs...
    + ./ArchiveBox.conf...

[+] Building main SQL index and running initial migrations...
    Operations to perform:
      Apply all migrations: admin, auth, contenttypes, core, sessions
    Running migrations:
    Applying contenttypes.0001_initial... OK
    Applying auth.0001_initial... OK
    Applying admin.0001_initial... OK
    Applying admin.0002_logentry_remove_auto_add... OK
    Applying admin.0003_logentry_add_action_flag_choices... OK
    Applying contenttypes.0002_remove_content_type_name... OK
    Applying auth.0002_alter_permission_name_max_length... OK
    Applying auth.0003_alter_user_email_max_length... OK
    Applying auth.0004_alter_user_username_opts... OK
    Applying auth.0005_alter_user_last_login_null... OK
    Applying auth.0006_require_contenttypes_0002... OK
    Applying auth.0007_alter_validators_add_error_messages... OK
    Applying auth.0008_alter_user_username_max_length... OK
    Applying auth.0009_alter_user_last_name_max_length... OK
    Applying auth.0010_alter_group_name_max_length... OK
    Applying auth.0011_update_proxy_permissions... OK
    Applying auth.0012_alter_user_first_name_max_length... OK
    Applying core.0001_initial... OK
    Applying core.0002_auto_20200625_1521... OK
    Applying core.0003_auto_20200630_1034... OK
    Applying core.0004_auto_20200713_1552... OK
    Applying core.0005_auto_20200728_0326... OK
    Applying core.0006_auto_20201012_1520... OK
    Applying core.0007_archiveresult... OK
    Applying core.0008_auto_20210105_1421... OK
    Applying core.0009_auto_20210216_1038... OK
    Applying core.0010_auto_20210216_1055... OK
    Applying core.0011_auto_20210216_1331... OK
    Applying core.0012_auto_20210216_1425... OK
    Applying core.0013_auto_20210218_0729... OK
    Applying core.0014_auto_20210218_0729... OK
    Applying core.0015_auto_20210218_0730... OK
    Applying core.0016_auto_20210218_1204... OK
    Applying core.0017_auto_20210219_0211... OK
    Applying core.0018_auto_20210327_0952... OK
    Applying core.0019_auto_20210401_0654... OK
    Applying core.0020_auto_20210410_1031... OK
    Applying sessions.0001_initial... OK

    √ ./index.sqlite3

[*] Checking links from indexes and archive folders (safe to Ctrl+C)...

[*] [2022-11-22 13:05:36] Writing 0 links to main index...
    √ ./index.sqlite3

----------------------------------------------------------------------
[√] Done. A new ArchiveBox collection was initialized (0 links).
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: disk I/O error

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/archivebox", line 33, in <module>
    sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')())
  File "/app/archivebox/cli/__init__.py", line 140, in main
    run_subcommand(
  File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/app/archivebox/cli/archivebox_init.py", line 43, in main
    init(
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/main.py", line 420, in init
    run_subcommand('setup', pwd=out_dir)
  File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/app/archivebox/cli/archivebox_setup.py", line 33, in main
    setup(
  File "/app/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/app/archivebox/main.py", line 906, in setup
    if not User.objects.filter(is_superuser=True).exists():
  File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 809, in exists
    return self.query.has_results(using=self.db)
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 535, in has_results
    return compiler.has_results()
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: disk I/O error

ArchiveBox version

ArchiveBox v0.6.2
Cpython Linux Linux-5.15.0-52-generic-x86_64-with-glibc2.28 x86_64
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.6.2          valid     /usr/local/bin/archivebox                                          
 √  PYTHON_BINARY         v3.9.5          valid     /usr/local/bin/python3.9                                           
 √  DJANGO_BINARY         v3.1.10         valid     /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py  
 √  CURL_BINARY           v7.64.0         valid     /usr/bin/curl                                                      
 √  WGET_BINARY           v1.20.1         valid     /usr/bin/wget                                                      
 √  NODE_BINARY           v15.14.0        valid     /usr/bin/node                                                      
 √  SINGLEFILE_BINARY     v0.3.16         valid     /node/node_modules/single-file/cli/single-file                     
 √  READABILITY_BINARY    v0.0.2          valid     /node/node_modules/readability-extractor/readability-extractor     
 √  MERCURY_BINARY        v1.0.0          valid     /node/node_modules/@postlight/mercury-parser/cli.js                
 √  GIT_BINARY            v2.20.1         valid     /usr/bin/git                                                       
 √  YOUTUBEDL_BINARY      v2021.04.26     valid     /usr/local/bin/youtube-dl                                          
 √  CHROME_BINARY         v90.0.4430.93   valid     /usr/bin/chromium                                                  
 √  RIPGREP_BINARY        v0.10.0         valid     /usr/bin/rg                                                        

[i] Source-code locations:
 √  PACKAGE_DIR           22 files        valid     /app/archivebox                                                    
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                          
 -  CUSTOM_TEMPLATES_DIR  -               disabled                                                                     

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                     
 -  COOKIES_FILE          -               disabled                                                                     


[i] Data locations:

Originally created by @rebane2001 on GitHub (Nov 22, 2022). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1052 #### Describe the bug Trying to set up ArchiveBox on a virtio mount results in an error (`sqlite3.OperationalError: disk I/O error`). #### Screenshots or log output fstab of the affected mount: ``` archive /mnt/archive 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0 ``` archivebox init logs: ```logs $ docker compose run archivebox init --setup [i] [2022-11-22 13:05:09] ArchiveBox v0.6.2: archivebox init --setup > /data [+] Initializing a new ArchiveBox v0.6.2 collection... ---------------------------------------------------------------------- [+] Building archive folder structure... + ./archive, ./sources, ./logs... + ./ArchiveBox.conf... [+] Building main SQL index and running initial migrations... Operations to perform: Apply all migrations: admin, auth, contenttypes, core, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying auth.0012_alter_user_first_name_max_length... OK Applying core.0001_initial... OK Applying core.0002_auto_20200625_1521... OK Applying core.0003_auto_20200630_1034... OK Applying core.0004_auto_20200713_1552... OK Applying core.0005_auto_20200728_0326... OK Applying core.0006_auto_20201012_1520... OK Applying core.0007_archiveresult... OK Applying core.0008_auto_20210105_1421... OK Applying core.0009_auto_20210216_1038... OK Applying core.0010_auto_20210216_1055... OK Applying core.0011_auto_20210216_1331... OK Applying core.0012_auto_20210216_1425... OK Applying core.0013_auto_20210218_0729... OK Applying core.0014_auto_20210218_0729... OK Applying core.0015_auto_20210218_0730... OK Applying core.0016_auto_20210218_1204... OK Applying core.0017_auto_20210219_0211... OK Applying core.0018_auto_20210327_0952... OK Applying core.0019_auto_20210401_0654... OK Applying core.0020_auto_20210410_1031... OK Applying sessions.0001_initial... OK √ ./index.sqlite3 [*] Checking links from indexes and archive folders (safe to Ctrl+C)... [*] [2022-11-22 13:05:36] Writing 0 links to main index... √ ./index.sqlite3 ---------------------------------------------------------------------- [√] Done. A new ArchiveBox collection was initialized (0 links). Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute return Database.Cursor.execute(self, query, params) sqlite3.OperationalError: disk I/O error The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/bin/archivebox", line 33, in <module> sys.exit(load_entry_point('archivebox', 'console_scripts', 'archivebox')()) File "/app/archivebox/cli/__init__.py", line 140, in main run_subcommand( File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore File "/app/archivebox/cli/archivebox_init.py", line 43, in main init( File "/app/archivebox/util.py", line 114, in typechecked_function return func(*args, **kwargs) File "/app/archivebox/main.py", line 420, in init run_subcommand('setup', pwd=out_dir) File "/app/archivebox/cli/__init__.py", line 80, in run_subcommand module.main(args=subcommand_args, stdin=stdin, pwd=pwd) # type: ignore File "/app/archivebox/cli/archivebox_setup.py", line 33, in main setup( File "/app/archivebox/util.py", line 114, in typechecked_function return func(*args, **kwargs) File "/app/archivebox/main.py", line 906, in setup if not User.objects.filter(is_superuser=True).exists(): File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 809, in exists return self.query.has_results(using=self.db) File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 535, in has_results return compiler.has_results() File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1126, in has_results return bool(self.execute_sql(SINGLE)) File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute return Database.Cursor.execute(self, query, params) django.db.utils.OperationalError: disk I/O error ``` #### ArchiveBox version  ```logs ArchiveBox v0.6.2 Cpython Linux Linux-5.15.0-52-generic-x86_64-with-glibc2.28 x86_64 IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep [i] Dependency versions: √ ARCHIVEBOX_BINARY v0.6.2 valid /usr/local/bin/archivebox √ PYTHON_BINARY v3.9.5 valid /usr/local/bin/python3.9 √ DJANGO_BINARY v3.1.10 valid /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py √ CURL_BINARY v7.64.0 valid /usr/bin/curl √ WGET_BINARY v1.20.1 valid /usr/bin/wget √ NODE_BINARY v15.14.0 valid /usr/bin/node √ SINGLEFILE_BINARY v0.3.16 valid /node/node_modules/single-file/cli/single-file √ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor √ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js √ GIT_BINARY v2.20.1 valid /usr/bin/git √ YOUTUBEDL_BINARY v2021.04.26 valid /usr/local/bin/youtube-dl √ CHROME_BINARY v90.0.4430.93 valid /usr/bin/chromium √ RIPGREP_BINARY v0.10.0 valid /usr/bin/rg [i] Source-code locations: √ PACKAGE_DIR 22 files valid /app/archivebox √ TEMPLATES_DIR 3 files valid /app/archivebox/templates - CUSTOM_TEMPLATES_DIR - disabled [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled - COOKIES_FILE - disabled [i] Data locations: ```

kerem

2026-03-15 00:00:17 +03:00

closed this issue
added the
status: needs followup
label

kerem commented

2026-03-15 00:00:33 +03:00

Author

Owner

@pirate commented on GitHub (Jun 13, 2023):

Sorry for the late response, unfortunately we don't support saving the SQLite DB on a filesystem that doesn't support FSYNC at the moment (as it can cause data loss when running multiple ArchiveBox commands at once).
You can find more information here: https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#filesystem

There is an undocumented footgun provided if you're willing to accept possible data loss when running multiple archivebox instances in parallel: set archivebox config ENFORCE_ATOMIC_WRITES=False https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config.py#L87.

Comment back here if you're still having problems and I can re-open the issue :)

@pirate commented on GitHub (Jun 13, 2023): Sorry for the late response, unfortunately we don't support saving the SQLite DB on a filesystem that doesn't support FSYNC at the moment (as it can cause data loss when running multiple ArchiveBox commands at once). You can find more information here: https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#filesystem There is an undocumented footgun provided if you're willing to accept possible data loss when running multiple archivebox instances in parallel: set `archivebox config ENFORCE_ATOMIC_WRITES=False` https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/config.py#L87. Comment back here if you're still having problems and I can re-open the issue :)

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#3680

No description provided.

Rows
Columns

[GH-ISSUE #1052] Question: Can't set up ArchiveBox DB on a virtio mount that doesn't support FSYNC (sqlite3.OperationalError: disk I/O error) #3680

Describe the bug

Screenshots or log output

ArchiveBox version

[GH-ISSUE #1052] Question: Can't set up ArchiveBox DB on a virtio mount that doesn't support FSYNC `(sqlite3.OperationalError: disk I/O error)` #3680