starred/ArchiveBox

Fork 0

mirror of https://github.com/ArchiveBox/ArchiveBox.git synced 2026-04-26 01:26:00 +03:00

[GH-ISSUE #945] Question: Mounting symlink'd index.sqlite3 on SSD when using docker-compose #3607

New issue

Closed

opened 2026-03-14 23:41:49 +03:00 by kerem · 6 comments

kerem commented

2026-03-14 23:41:49 +03:00

Owner

Originally created by @akhilleusuggo on GitHub (Mar 15, 2022).
Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/945

Hello,

I've tried many ways to make it work, but I can't achieve to have the database hosted outside of the /data directory.
I'm running the project on ZFS (hard drives), the performance is O.K, but it start to get slow on the DB side, and when trying to add multiple links error 500 starts occurring.
You suggest running the database at the least on SSD https://github.com/ArchiveBox/ArchiveBox#storage-requirements , but doesn't seem to work with docker-compose, or I'm unable to mount it properly.

version: '2.4'   # '3.9' or greater also works

services:
    archivebox:
        # build: .                              # for developers working on archivebox
        image: ${DOCKER_IMAGE:-archivebox/archivebox:master}
        command: server --quick-init 0.0.0.0:8000
        ports:
            - 8000:8000
        environment:
            - ALLOWED_HOSTS=*                   # add any config options you want as env vars
            - SAVE_MEDIA=False
            #- MEDIA_MAX_SIZE=750m
            # - SEARCH_BACKEND_ENGINE=sonic     # uncomment these if you enable sonic below
            # - SEARCH_BACKEND_HOST_NAME=sonic
            # - SEARCH_BACKEND_PASSWORD=SecretPassword
        volumes:
            - /docker/archivebox/data:/data

My home directory is an NVMe SSD, I would like to have the index.sqlite3 on /home/user/data/index.sqlite3

Tried symbolic link, but does fail. Compose doesn't start, looping errors.
Tried to add a volume like this, but keeps recreating the database from scratch.

 volumes:
            - /docker/archivebox/data:/data
            - /home/user/data/index.sqlite3:/data/index.sqlite3

Any proper way of mounting it?

Originally created by @akhilleusuggo on GitHub (Mar 15, 2022). Original GitHub issue: https://github.com/ArchiveBox/ArchiveBox/issues/945 Hello, I've tried many ways to make it work, but I can't achieve to have the database hosted outside of the /data directory. I'm running the project on ZFS (hard drives), the performance is O.K, but it start to get slow on the DB side, and when trying to add multiple links error 500 starts occurring. You suggest running the database at the least on SSD https://github.com/ArchiveBox/ArchiveBox#storage-requirements , but doesn't seem to work with docker-compose, or I'm unable to mount it properly. ``` version: '2.4' # '3.9' or greater also works services: archivebox: # build: . # for developers working on archivebox image: ${DOCKER_IMAGE:-archivebox/archivebox:master} command: server --quick-init 0.0.0.0:8000 ports: - 8000:8000 environment: - ALLOWED_HOSTS=* # add any config options you want as env vars - SAVE_MEDIA=False #- MEDIA_MAX_SIZE=750m # - SEARCH_BACKEND_ENGINE=sonic # uncomment these if you enable sonic below # - SEARCH_BACKEND_HOST_NAME=sonic # - SEARCH_BACKEND_PASSWORD=SecretPassword volumes: - /docker/archivebox/data:/data ``` My home directory is an NVMe SSD, I would like to have the `index.sqlite3` on `/home/user/data/index.sqlite3` Tried symbolic link, but does fail. Compose doesn't start, looping errors. Tried to add a volume like this, but keeps recreating the database from scratch. ``` volumes: - /docker/archivebox/data:/data - /home/user/data/index.sqlite3:/data/index.sqlite3 ``` Any proper way of mounting it?

kerem closed this issue

2026-03-14 23:41:55 +03:00

kerem commented

2026-03-14 23:42:00 +03:00

Author

Owner

@akhilleusuggo commented on GitHub (Mar 15, 2022):

I've managed to fix my issue. Seems like docker is not very friendly with symbolic links.

The mount folder/files must be identical of the docker.

In this example, I had to create a /data folder on the host machine. Copy to it the index.sqlite3
Expose the index.sqlite3 in the docker container volumes.

/data/index.sqlite3:/data/index.sqlite3

Not sure why*, but the script keeps creating a dummy file index.sqlite3

@akhilleusuggo commented on GitHub (Mar 15, 2022): I've managed to fix my issue. Seems like docker is not very friendly with symbolic links. The mount folder/files must be identical of the docker. In this example, I had to create a `/data` folder on the host machine. Copy to it the index.sqlite3 Expose the index.sqlite3 in the docker container volumes. `/data/index.sqlite3:/data/index.sqlite3` Not sure why*, but the script keeps creating a dummy file `index.sqlite3`

kerem commented

2026-03-14 23:42:05 +03:00

Author

Owner

@pirate commented on GitHub (Mar 15, 2022):

Correct, docker doesn't work to mount symbolic links as volumes.
If the mounted file is not valid inside docker it will try to overwrite it an create that empty index.sqlite3 you're seeing.

Hard links should work however, or you can mount the symlink destination as a volume as well (at the same path inside and outside of docker) and then it should work.

@pirate commented on GitHub (Mar 15, 2022): Correct, docker doesn't work to mount symbolic links as volumes. If the mounted file is not valid inside docker it will try to overwrite it an create that empty index.sqlite3 you're seeing. Hard links should work however, or you can mount the symlink destination as a volume as well (at the same path inside and outside of docker) and then it should work.

kerem commented

2026-03-14 23:42:11 +03:00

Author

Owner

@akhilleusuggo commented on GitHub (Mar 15, 2022):

Could you explain what do you mean by

you can mount the symlink destination as a volume as well (at the same path inside and outside of docker)

If you could provide an example I'd appreciate it.

@akhilleusuggo commented on GitHub (Mar 15, 2022): Could you explain what do you mean by > you can mount the symlink destination as a volume as well (at the same path inside and outside of docker) If you could provide an example I'd appreciate it.

kerem commented

2026-03-14 23:42:16 +03:00

Author

Owner

@pirate commented on GitHub (Mar 16, 2022):

For example if you have this folder layout on the host:

~/archivebox
    data/
        index.sqlite3 -> /media/someSSD/archivebox/index.sqlite (symlink)
        archive/
            1234566778/
            ...
        ...

Then you would need to mount it like so in docker-compose.yml:

services:
    archivebox:
        ...
        volumes:
            - ./data:/data
            - /media/someSSD/archivebox/index.sqlite:/media/someSSD/archivebox/index.sqlite

(note the symlink path must be mounted to the same path as it is on the host inside docker)

@pirate commented on GitHub (Mar 16, 2022): For example if you have this folder layout on the host: ``` ~/archivebox data/ index.sqlite3 -> /media/someSSD/archivebox/index.sqlite (symlink) archive/ 1234566778/ ... ... ``` Then you would need to mount it like so in `docker-compose.yml`: ```yaml services: archivebox: ... volumes: - ./data:/data - /media/someSSD/archivebox/index.sqlite:/media/someSSD/archivebox/index.sqlite ``` (note the symlink path must be mounted to the same path as it is on the host inside docker)

kerem commented

2026-03-14 23:42:21 +03:00

Author

Owner

@akhilleusuggo commented on GitHub (Mar 16, 2022):

Ok, I see what you mean. Like a reverse symlink.
Thank you, I'll try this way out.

@akhilleusuggo commented on GitHub (Mar 16, 2022): Ok, I see what you mean. Like a reverse symlink. Thank you, I'll try this way out.

kerem commented

2026-03-14 23:42:26 +03:00

Author

Owner

@akhilleusuggo commented on GitHub (Apr 13, 2022):

@pirate I don't want to reopen the issue, but your last example logically works, but when it comes to running the database, how would I do that?

How can I tell Archivebox that the database (index.sqlite3) is mounted on /media/someSSD/archivebox/index.sqlite and not on the default location data/index.sqlite3

Just by creating a symbolic link, doesn't mean the data will be written to the location of the symlink. Will still be written on the default /data/index.sqlite3 witch I have on spinning drives with ZFS (compression/dedup ON), and you know performance is horrible compared to without.

Your example maybe was meant to fix the docker symlink issue; but not Archivebox issue.

If more than 1 project has a database that requires mounting on /data/index.sqlite3, this solution will not work.
I'm not saying that archivebox issue, but would be nice if we had a variable to point where the database is located.

I know that maybe the last resort would that each one would be rebuilding the image to reflect the location where he wants to store the db.

@akhilleusuggo commented on GitHub (Apr 13, 2022): @pirate I don't want to reopen the issue, but your last example logically works, but when it comes to running the database, how would I do that? How can I tell Archivebox that the database (index.sqlite3) is mounted on `/media/someSSD/archivebox/index.sqlite` and not on the default location `data/index.sqlite3` Just by creating a symbolic link, doesn't mean the data will be written to the location of the symlink. Will still be written on the default /data/index.sqlite3 witch I have on spinning drives with ZFS (compression/dedup ON), and you know performance is horrible compared to without. Your example maybe was meant to fix the docker symlink issue; but not Archivebox issue. If more than 1 project has a database that requires mounting on /data/index.sqlite3, this solution will not work. I'm not saying that archivebox issue, but would be nice if we had a variable to point where the database is located. I know that maybe the last resort would that each one would be rebuilding the image to reflect the location where he wants to store the db.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/ArchiveBox#3607

No description provided.

Rows
Columns