starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-25 17:16:00 +03:00

[GH-ISSUE #894] Question: How to run AB on localhost but store data on NAS? #553

New issue

Closed

opened 2026-03-01 14:44:32 +03:00 by kerem · 5 comments

kerem commented

2026-03-01 14:44:32 +03:00

Owner

Originally created by @iwconfig on GitHub (Nov 24, 2021).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/894

Hello!

I'm using docker-compose. Following the title if this issue: I first tried changing the whole data dir to a mounted path pointing to my NAS, but I got this error:

  archivebox_1  | [X] OSError: Failed to write /data/ArchiveBox.conf with fcntl.F_FULLFSYNC. ([Errno 22] Invalid argument)
  archivebox_1  |     You can store the archive/ subfolder on a hard drive or network share that doesn't support support syncronous writes,
  archivebox_1  |     but the main folder containing the index.sqlite3 and ArchiveBox.conf files must be on a filesystem that supports FSYNC.

I cannot figure out how to separate data/archive/ from the rest as I wish to store this on my NAS. I tried symlinking it but it just complains that data/archive/ already exists.

I would prefer to have as much data on my NAS so all or any of data/{logs,sonic,sources}/ as well.

How do I setup this with docker-compose?

Thank you kindly! in advance.

Originally created by @iwconfig on GitHub (Nov 24, 2021). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/894 Hello! I'm using docker-compose. Following the title if this issue: I first tried changing the whole data dir to a mounted path pointing to my NAS, but I got this error: archivebox_1 | [X] OSError: Failed to write /data/ArchiveBox.conf with fcntl.F_FULLFSYNC. ([Errno 22] Invalid argument) archivebox_1 | You can store the archive/ subfolder on a hard drive or network share that doesn't support support syncronous writes, archivebox_1 | but the main folder containing the index.sqlite3 and ArchiveBox.conf files must be on a filesystem that supports FSYNC. I cannot figure out how to separate `data/archive/` from the rest as I wish to store this on my NAS. I tried symlinking it but it just complains that `data/archive/` already exists. I would prefer to have as much data on my NAS so all or any of `data/{logs,sonic,sources}/` as well. How do I setup this with docker-compose? Thank you kindly! in advance.

kerem closed this issue

2026-03-01 14:44:32 +03:00

kerem commented

2026-03-01 14:44:32 +03:00

Author

Owner

@pirate commented on GitHub (Nov 26, 2021):

Symlink works but you have to mount the destination of the symlink in docker-compose as a volume as well (to the same location the symlink is pointing to, except inside the container).

@pirate commented on GitHub (Nov 26, 2021): Symlink works but you have to mount the destination of the symlink in docker-compose as a volume as well (to the same location the symlink is pointing to, except inside the container).

kerem commented

2026-03-01 14:44:33 +03:00

Author

Owner

@iwconfig commented on GitHub (Nov 26, 2021):

I'm sorry, I don't understand this. Can you give me an example?

Since ArchiveBox.conf and index.sqlite3 needs to be on host and not on NAS, how do I separate them from the rest?

I've tried many different variations, for example:

volumes:
    - ./data:/data
    - /mnt/arkivet/ArchiveBox-data/archive:/mnt/arkivet/ArchiveBox-data/archive
    - /mnt/arkivet/ArchiveBox-data/sources:/mnt/arkivet/ArchiveBox-data/sources
    - /mnt/arkivet/ArchiveBox-data/logs:/mnt/arkivet/ArchiveBox-data/logs

But this doesn't work as the folders are created in host ./data. I don't know when I should create my symlinks either because AB complains if a folder (symlink) already exist or creates the folders before I have a chance to create my symlinks. It doesn't matter if I remove folder(s) and then create my symlink(s), it still complains about what I believe is FSYNC.

What am I missing?

Traceback (most recent call last):
  File "/app/archivebox/system.py", line 90, in atomic_write
    f.write(contents)
  File "/usr/local/lib/python3.9/contextlib.py", line 124, in __exit__
    next(self.gen)
  File "/app/archivebox/vendor/atomicwrites.py", line 172, in _open
    self.commit(f)
  File "/app/archivebox/vendor/atomicwrites.py", line 205, in commit
    replace_atomic(f.name, self._path)
  File "/app/archivebox/vendor/atomicwrites.py", line 99, in replace_atomic
    return _replace_atomic(src, dst)
  File "/app/archivebox/vendor/atomicwrites.py", line 56, in _replace_atomic
    _sync_directory(os.path.normpath(os.path.dirname(dst)))
  File "/app/archivebox/vendor/atomicwrites.py", line 50, in _sync_directory
    _proper_fsync(fd)
OSError: [Errno 22] Invalid argument

@iwconfig commented on GitHub (Nov 26, 2021): I'm sorry, I don't understand this. Can you give me an example? Since `ArchiveBox.conf` and `index.sqlite3` needs to be on host and not on NAS, how do I separate them from the rest? I've tried many different variations, for example: volumes: - ./data:/data - /mnt/arkivet/ArchiveBox-data/archive:/mnt/arkivet/ArchiveBox-data/archive - /mnt/arkivet/ArchiveBox-data/sources:/mnt/arkivet/ArchiveBox-data/sources - /mnt/arkivet/ArchiveBox-data/logs:/mnt/arkivet/ArchiveBox-data/logs But this doesn't work as the folders are created in host `./data`. I don't know when I should create my symlinks either because AB complains if a folder (symlink) already exist or creates the folders before I have a chance to create my symlinks. It doesn't matter if I remove folder(s) and then create my symlink(s), it still complains about what I believe is FSYNC. What am I missing? ``` Traceback (most recent call last): File "/app/archivebox/system.py", line 90, in atomic_write f.write(contents) File "/usr/local/lib/python3.9/contextlib.py", line 124, in __exit__ next(self.gen) File "/app/archivebox/vendor/atomicwrites.py", line 172, in _open self.commit(f) File "/app/archivebox/vendor/atomicwrites.py", line 205, in commit replace_atomic(f.name, self._path) File "/app/archivebox/vendor/atomicwrites.py", line 99, in replace_atomic return _replace_atomic(src, dst) File "/app/archivebox/vendor/atomicwrites.py", line 56, in _replace_atomic _sync_directory(os.path.normpath(os.path.dirname(dst))) File "/app/archivebox/vendor/atomicwrites.py", line 50, in _sync_directory _proper_fsync(fd) OSError: [Errno 22] Invalid argument ```

kerem commented

2026-03-01 14:44:33 +03:00

Author

Owner

@cmuench commented on GitHub (Dec 10, 2021):

@iwconfig I was able to share my archive directory on my Synology NAS.
I mouted the archive directory directly from NFS.

version: '2.4'

services:
    archivebox:        
        image: ${DOCKER_IMAGE:-archivebox/archivebox:master}
        command: server --quick-init 0.0.0.0:8000
        restart: unless-stopped
        environment:
            - ALLOWED_HOSTS=*
            - MEDIA_MAX_SIZE=750m
        volumes:
            - ./volumes/data:/data
            - archivebox-archive:/data/archive

volumes:
  archivebox-archive:
      driver: local
      driver_opts:
        type: "nfs"
        o: "nfsvers=4,addr=192.168.1.12,rw"
        device: ":/volume1/archivebox/archive"

Difference is that I used docker to create the mount instead of fstab. Hope that also works on your side.

@cmuench commented on GitHub (Dec 10, 2021): @iwconfig I was able to share my archive directory on my Synology NAS. I mouted the archive directory directly from NFS. ```yaml version: '2.4' services: archivebox: image: ${DOCKER_IMAGE:-archivebox/archivebox:master} command: server --quick-init 0.0.0.0:8000 restart: unless-stopped environment: - ALLOWED_HOSTS=* - MEDIA_MAX_SIZE=750m volumes: - ./volumes/data:/data - archivebox-archive:/data/archive volumes: archivebox-archive: driver: local driver_opts: type: "nfs" o: "nfsvers=4,addr=192.168.1.12,rw" device: ":/volume1/archivebox/archive" ``` Difference is that I used docker to create the mount instead of fstab. Hope that also works on your side.

kerem commented

2026-03-01 14:44:33 +03:00

Author

Owner

@iwconfig commented on GitHub (Dec 10, 2021):

Thank you so much! I got it working now 😃 I just had one issue with permissions inside the docker container, so I had to do

docker exec -it archivebox_archivebox_1 chmod -R 777 /data/archive

Awesome! Again, thank you!

@pirate Maybe this should be mentioned in the documentation? If it already is, I'm sorry, I haven't seen it.

@iwconfig commented on GitHub (Dec 10, 2021): Thank you so much! I got it working now :smiley: I just had one issue with permissions inside the docker container, so I had to do ``` docker exec -it archivebox_archivebox_1 chmod -R 777 /data/archive ``` Awesome! Again, thank you! @pirate Maybe this should be mentioned in the documentation? If it already is, I'm sorry, I haven't seen it.

kerem commented

2026-03-01 14:44:33 +03:00

Author

Owner

@pirate commented on GitHub (Apr 12, 2022):

Note I've added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.

@pirate commented on GitHub (Apr 12, 2022): Note I've added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting Contributions/suggestions welcome there.

kerem referenced this issue

2026-03-01 17:54:23 +03:00

[GH-ISSUE #553] queuing for add #1859

kerem referenced this issue

2026-03-14 22:28:08 +03:00

[GH-ISSUE #553] queuing for add #3371

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#553

No description provided.

Rows
Columns