starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 17:16:00 +03:00

[GH-ISSUE #1521] Bug: 0.7.2 Example docker-compose.yml sonic config download races with bind mounted config file and errors out #2413

New issue

Closed

opened 2026-03-01 17:58:52 +03:00 by kerem · 7 comments

kerem commented

2026-03-01 17:58:52 +03:00

Owner

Originally created by @agowa on GitHub (Sep 22, 2024).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1521

Describe the bug

The provided configuration for sonic within the easy setup docker-compose.yml is invalid.

Steps to reproduce

Enable sonic within the easy setup docker-compose.yml and run docker compose up

Screenshots or log output

sonic-1                 | thread 'main' panicked at src/config/reader.rs:23:14:
sonic-1                 | cannot read config file: Os { code: 21, kind: IsADirectory, message: "Is a directory" }
sonic-1                 | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
sonic-1 exited with code 101

ArchiveBox version

[user@PC-001 archivebox]$ docker compose run archivebox version
>>>> Executing external compose provider "/usr/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<<

WARN[0000] Found orphan containers ([html_spider-archivebox-run-f35b9dbf9381 html_spider-archivebox-run-a6b700077112 html_spider-archivebox-run-ead5faa5f13e html_spider-archivebox-run-9e3402b5cd9b html_spider-archivebox-run-ba456959b5e5 html_spider-archivebox-run-a45c3aa6693b html_spider-archivebox-run-a662effa2c75 html_spider-archivebox-run-5741b0d8132e html_spider-archivebox-run-cba21b6a84b5 html_spider-archivebox-run-b49df6d7d999 html_spider-archivebox-run-f8e29ecf079b]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 
0.7.2
ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:47:02 1713998822
IN_DOCKER=True IN_QEMU=False ARCH=x86_64 OS=Linux PLATFORM=Linux-6.10.10-arch1-1-x86_64-with-glibc2.36 PYTHON=Cpython
FS_ATOMIC=True FS_REMOTE=True FS_USER=1000:1000 FS_PERMS=644
DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND=sonic LDAP=False

[i] Dependency versions:
 √  PYTHON_BINARY         v3.11.9         valid     /usr/local/bin/python3.11                                                   
 √  SQLITE_BINARY         v2.6.0          valid     /usr/local/lib/python3.11/sqlite3/dbapi2.py                                 
 √  DJANGO_BINARY         v3.1.14         valid     /usr/local/lib/python3.11/site-packages/django/__init__.py                  
 √  ARCHIVEBOX_BINARY     v0.7.2          valid     /usr/local/bin/archivebox                                                   

 √  CURL_BINARY           v8.5.0          valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.21.3         valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v20.12.2        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v1.1.46         valid     /app/node_modules/single-file-cli/single-file                               
 √  READABILITY_BINARY    v0.0.11         valid     /app/node_modules/readability-extractor/readability-extractor               
 √  MERCURY_BINARY        v1.0.0          valid     /app/node_modules/@postlight/parser/cli.js                                  
 √  GIT_BINARY            v2.39.2         valid     /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2023.12.30     valid     /usr/local/bin/yt-dlp                                                       
 √  CHROME_BINARY         v124.0.6367.29  valid     /usr/bin/chromium-browser                                                   
 √  RIPGREP_BINARY        v13.0.0         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /app/archivebox                                                             
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                                   
 -  CUSTOM_TEMPLATES_DIR  -               disabled  None                                                                        

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled  None                                                                        
 -  COOKIES_FILE          -               disabled  None                                                                        

[i] Data locations:
 √  OUTPUT_DIR            7 files @       valid     /data                                                                       
 √  SOURCES_DIR           3 files         valid     ./sources                                                                   
 √  LOGS_DIR              1 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           122 files       valid     ./archive                                                                   
 √  CONFIG_FILE           81.0 Bytes      valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             784.0 KB        valid     ./index.sqlite3

Originally created by @agowa on GitHub (Sep 22, 2024). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/1521 #### Describe the bug The provided configuration for sonic within the easy setup docker-compose.yml is invalid. #### Steps to reproduce Enable sonic within the easy setup docker-compose.yml and run `docker compose up` #### Screenshots or log output ``` sonic-1 | thread 'main' panicked at src/config/reader.rs:23:14: sonic-1 | cannot read config file: Os { code: 21, kind: IsADirectory, message: "Is a directory" } sonic-1 | note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace sonic-1 exited with code 101 ``` #### ArchiveBox version ```logs [user@PC-001 archivebox]$ docker compose run archivebox version >>>> Executing external compose provider "/usr/bin/docker-compose". Please see podman-compose(1) for how to disable this message. <<<< WARN[0000] Found orphan containers ([html_spider-archivebox-run-f35b9dbf9381 html_spider-archivebox-run-a6b700077112 html_spider-archivebox-run-ead5faa5f13e html_spider-archivebox-run-9e3402b5cd9b html_spider-archivebox-run-ba456959b5e5 html_spider-archivebox-run-a45c3aa6693b html_spider-archivebox-run-a662effa2c75 html_spider-archivebox-run-5741b0d8132e html_spider-archivebox-run-cba21b6a84b5 html_spider-archivebox-run-b49df6d7d999 html_spider-archivebox-run-f8e29ecf079b]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up. 0.7.2 ArchiveBox v0.7.2 COMMIT_HASH=315c9f3 BUILD_TIME=2024-04-24 22:47:02 1713998822 IN_DOCKER=True IN_QEMU=False ARCH=x86_64 OS=Linux PLATFORM=Linux-6.10.10-arch1-1-x86_64-with-glibc2.36 PYTHON=Cpython FS_ATOMIC=True FS_REMOTE=True FS_USER=1000:1000 FS_PERMS=644 DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND=sonic LDAP=False [i] Dependency versions: √ PYTHON_BINARY v3.11.9 valid /usr/local/bin/python3.11 √ SQLITE_BINARY v2.6.0 valid /usr/local/lib/python3.11/sqlite3/dbapi2.py √ DJANGO_BINARY v3.1.14 valid /usr/local/lib/python3.11/site-packages/django/__init__.py √ ARCHIVEBOX_BINARY v0.7.2 valid /usr/local/bin/archivebox √ CURL_BINARY v8.5.0 valid /usr/bin/curl √ WGET_BINARY v1.21.3 valid /usr/bin/wget √ NODE_BINARY v20.12.2 valid /usr/bin/node √ SINGLEFILE_BINARY v1.1.46 valid /app/node_modules/single-file-cli/single-file √ READABILITY_BINARY v0.0.11 valid /app/node_modules/readability-extractor/readability-extractor √ MERCURY_BINARY v1.0.0 valid /app/node_modules/@postlight/parser/cli.js √ GIT_BINARY v2.39.2 valid /usr/bin/git √ YOUTUBEDL_BINARY v2023.12.30 valid /usr/local/bin/yt-dlp √ CHROME_BINARY v124.0.6367.29 valid /usr/bin/chromium-browser √ RIPGREP_BINARY v13.0.0 valid /usr/bin/rg [i] Source-code locations: √ PACKAGE_DIR 23 files valid /app/archivebox √ TEMPLATES_DIR 3 files valid /app/archivebox/templates - CUSTOM_TEMPLATES_DIR - disabled None [i] Secrets locations: - CHROME_USER_DATA_DIR - disabled None - COOKIES_FILE - disabled None [i] Data locations: √ OUTPUT_DIR 7 files @ valid /data √ SOURCES_DIR 3 files valid ./sources √ LOGS_DIR 1 files valid ./logs √ ARCHIVE_DIR 122 files valid ./archive √ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf √ SQL_INDEX 784.0 KB valid ./index.sqlite3 ```

kerem closed this issue

2026-03-01 17:58:52 +03:00

kerem commented

2026-03-01 17:58:53 +03:00

Author

Owner

@agowa commented on GitHub (Sep 22, 2024):

the issue here is that docker-compose will create a folder "sonic.cfg" in the projects directory. See https://docs.docker.com/reference/compose-file/services/#long-syntax-5
The default is bind and for it when there is no file on the host it'll create a directory. The only other option is to disable this using the create_host_path option, but then our config would most likely stay inside the container.
The desired option of copying the sonic.cfg from within the container to the host if absent is only available for volumes. But volumes of type we would also get a folder and it would be way more difficult to have it point to a file in the project directory.

The most straight forward way would be to have the entrypoint/startup script of the sonic container check if /etc/sonic.cfg is empty and if it is overwrite it with the default config.

In k8s an alternative solution would be to use an initcontainer to do this but docker-compose doesn't have them. But in k8s we could also put this config file into a ConfigMap object. Interestingly docker-compose has also a "configs" section but sadly it does not allow to use the file within the container as the default. The "file" option of it only allows for using a file within the project directory as the default. See https://github.com/compose-spec/compose-spec/blob/main/08-configs.md and https://github.com/compose-spec/compose-spec/blob/main/spec.md Also the configs section in docker-compose is read-only I think...

Also when someone is using the podman-compose wrapper also this limitation can be observed: https://github.com/containers/podman-compose/issues/864
(but even with the "real" docker-compose then even though the dockerfile_inline is executed it still doesn't create the sonic.cfg as a file. I still did get it as a folder...

Either way the sonic section needs to be changed. For now the simplest workaround is to run curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/etc/sonic.cfg' > sonic.cfg within the directory of the docker-compose.yml file.

@agowa commented on GitHub (Sep 22, 2024): the issue here is that docker-compose will create a folder "sonic.cfg" in the projects directory. See https://docs.docker.com/reference/compose-file/services/#long-syntax-5 The default is bind and for it when there is no file on the host it'll create a directory. The only other option is to disable this using the create_host_path option, but then our config would most likely stay inside the container. The desired option of copying the sonic.cfg from within the container to the host if absent is only available for volumes. But volumes of type we would also get a folder and it would be way more difficult to have it point to a file in the project directory. The most straight forward way would be to have the entrypoint/startup script of the sonic container check if /etc/sonic.cfg is empty and if it is overwrite it with the default config. In k8s an alternative solution would be to use an initcontainer to do this but docker-compose doesn't have them. But in k8s we could also put this config file into a ConfigMap object. Interestingly docker-compose has also a "configs" section but sadly it does not allow to use the file within the container as the default. The "file" option of it only allows for using a file within the project directory as the default. See https://github.com/compose-spec/compose-spec/blob/main/08-configs.md and https://github.com/compose-spec/compose-spec/blob/main/spec.md Also the configs section in docker-compose is read-only I think... Also when someone is using the podman-compose wrapper also this limitation can be observed: https://github.com/containers/podman-compose/issues/864 (but even with the "real" docker-compose then even though the dockerfile_inline is executed it still doesn't create the sonic.cfg as a file. I still did get it as a folder... Either way the sonic section needs to be changed. For now the simplest workaround is to run `curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/etc/sonic.cfg' > sonic.cfg` within the directory of the docker-compose.yml file.

kerem commented

2026-03-01 17:58:53 +03:00

Author

Owner

@pirate commented on GitHub (Sep 22, 2024):

Fixed: github.com/ArchiveBox/ArchiveBox@212280859c

It's just a race between the host bind and the container downloader, if host binds first then it's a folder, if container downloads it first then it's a file. I just commented out the bind in the example config. :ro ensures the container will never overwrite a file provided by the host. There's no point mounting the default config outside the container, if the user wants to modify it they should just download it manually and bind mount it at that point.

@pirate commented on GitHub (Sep 22, 2024): Fixed: https://github.com/ArchiveBox/ArchiveBox/commit/212280859cf0b1ba378f0a4a3afda1c06961fcb9 It's just a race between the host bind and the container downloader, if host binds first then it's a folder, if container downloads it first then it's a file. I just commented out the bind in the example config. `:ro` ensures the container will never overwrite a file provided by the host. There's no point mounting the default config outside the container, if the user wants to modify it they should just download it manually and bind mount it at that point.

kerem commented

2026-03-01 17:58:53 +03:00

Author

Owner

@agowa commented on GitHub (Sep 22, 2024):

Hi, I think I just now found a better way to fix this. In the attached PR.

@agowa commented on GitHub (Sep 22, 2024): Hi, I think I just now found a better way to fix this. In the attached PR.

kerem commented

2026-03-01 17:58:53 +03:00

Author

Owner

@satonotdead commented on GitHub (Oct 7, 2024):

It doesn't fix the issue for me, I'm having permissions errors from a few months and I can't figure it out.

PID/GUID don't work. Tried archivebox/root/myuser as user owner or group, nothing helps. Original docker-compose show archivebox-1 exited with code 0 and all the ticket related are closed.

It should be very valuable to keep issues opened until their get confirmations to get closed.

@satonotdead commented on GitHub (Oct 7, 2024): It doesn't fix the issue for me, I'm having permissions errors from a few months and I can't figure it out. PID/GUID don't work. Tried archivebox/root/myuser as user owner or group, nothing helps. Original docker-compose show `archivebox-1 exited with code 0` and all the ticket related are closed. It should be very valuable to keep issues opened until their get confirmations to get closed.

kerem commented

2026-03-01 17:58:53 +03:00

Author

Owner

@pirate commented on GitHub (Oct 7, 2024):

@satoshinotdead there any many potential causes of permissions issues, I wouldn't be too sure that your issue is the same as the one here. Please open a new issue and share your full docker-compose.yml and docker compose run archivebox version output (even if it fails to start) + screenshot of any error output.

@pirate commented on GitHub (Oct 7, 2024): @satoshinotdead there any many potential causes of permissions issues, I wouldn't be too sure that your issue is the same as the one here. Please open a new issue and share your full `docker-compose.yml` and `docker compose run archivebox version` output (even if it fails to start) + screenshot of any error output.

kerem commented

2026-03-01 17:58:53 +03:00

Author

Owner

@satonotdead commented on GitHub (Oct 7, 2024):

Thanks, I will look into it because tweaking the paths on sonic.cfg (with template updated ones) solves this issue but I'm still with permission errors.

@satonotdead commented on GitHub (Oct 7, 2024): Thanks, I will look into it because tweaking the paths on sonic.cfg (with template updated ones) solves this issue but I'm still with permission errors.

kerem commented

2026-03-01 17:58:54 +03:00

Author

Owner

@pirate commented on GitHub (Nov 13, 2024):

I made a new amd64/arm64 archivebox/sonic container that bundles the default config file, so dockerfile_inline is no longer used in docker-compose.yml and podman should work now:

@pirate commented on GitHub (Nov 13, 2024): I made a new `amd64`/`arm64` `archivebox/sonic` container that bundles the default config file, so `dockerfile_inline` is no longer used in `docker-compose.yml` and podman should work now: - https://github.com/ArchiveBox/ArchiveBox/commit/6448968952b3d7f94c18ce1da4bee305c7e05f86 - https://hub.docker.com/r/archivebox/sonic

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#2413

No description provided.

Rows
Columns