starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00

[GH-ISSUE #2055] High memory usage with big files. #1041

New issue

Open

opened 2026-03-04 01:50:53 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:50:53 +03:00

Owner

Originally created by @Messiah93 on GitHub (Nov 14, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2055

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

s3fs --version

Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

pacman -Qi fuse

Name : fuse2
Version : 2.9.9-4
Description : A library that makes it possible to implement a filesystem in
a userspace program.
Architecture : x86_64
URL : https://github.com/libfuse/libfuse
Licenses : GPL2
Groups : None
Provides : fuse=2.9.9
Depends On : glibc fuse-common
Optional Deps : None
Required By : s3fs-fuse
Optional For : grub
Conflicts With : fuse
Replaces : fuse
Installed Size : 458.21 KiB
Packager : Felix Yan felixonmars@archlinux.org
Build Date : Sat 16 May 2020 12:21:51 PM CEST
Install Date : Sun 13 Nov 2022 10:46:04 AM CET
Install Reason : Installed as a dependency for another package
Install Script : No
Validated By : Signature

Kernel information (uname -r)

uname -r

5.15.78-1-lts

GNU/Linux Distribution, if applicable (cat /etc/os-release)

cat /etc/os-release

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

s3fs command line used, if applicable

s3fs backup-extended-us1 /media/backup-extended-us1 -o allow_other -o use_path_request_style -o passwd_file=/etc/passwd-s3fs -o url=https://usc1.contabostorage.com/

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

VPS: 4vCore, 4Go RAM, 500Go SSD, 200mbits bandwidth @ Archlinux.
My s3fs bucket is empty at this point.
The file "m93_cpre_2022-11-13_-_00h00.tgz" size is ~7go.

Before mounting bucket with s3fs, RAM usage is about ~20%.
After mounting bucket with s3fs, RAM usage is about ~22%.

cp "/media/backup/m93_cpre_2022-11-12_-00h00.tgz" "/media/backup-extended-us1/m93_cpre_2022-11-12-_00h00.tgz"

After 1 minute, RAM usage is about 99%.
After 2 minutes, VPS take ~1 minute to connect by SSH with new connection.
After 4-5 minutes, VPS is no longer responding for 24 hours+, I need to hard restart it.

CPU usage is very low, I/O too, bandwidth same... I don't know if this high memory usage is normal, or if I do something wrong, i'm new to s3fs since 2 days.

Thank's.

Originally created by @Messiah93 on GitHub (Nov 14, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2055 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) s3fs --version Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) pacman -Qi fuse Name : fuse2 Version : 2.9.9-4 Description : A library that makes it possible to implement a filesystem in a userspace program. Architecture : x86_64 URL : https://github.com/libfuse/libfuse Licenses : GPL2 Groups : None Provides : fuse=2.9.9 Depends On : glibc fuse-common Optional Deps : None Required By : s3fs-fuse Optional For : grub Conflicts With : fuse Replaces : fuse Installed Size : 458.21 KiB Packager : Felix Yan <felixonmars@archlinux.org> Build Date : Sat 16 May 2020 12:21:51 PM CEST Install Date : Sun 13 Nov 2022 10:46:04 AM CET Install Reason : Installed as a dependency for another package Install Script : No Validated By : Signature #### Kernel information (uname -r) uname -r 5.15.78-1-lts #### GNU/Linux Distribution, if applicable (cat /etc/os-release) cat /etc/os-release NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://bugs.archlinux.org/" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo #### s3fs command line used, if applicable ``` s3fs backup-extended-us1 /media/backup-extended-us1 -o allow_other -o use_path_request_style -o passwd_file=/etc/passwd-s3fs -o url=https://usc1.contabostorage.com/ ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue VPS: 4vCore, 4Go RAM, 500Go SSD, 200mbits bandwidth @ Archlinux. My s3fs bucket is empty at this point. The file "m93_cpre_2022-11-13_-_00h00.tgz" size is ~7go. Before mounting bucket with s3fs, RAM usage is about ~20%. After mounting bucket with s3fs, RAM usage is about ~22%. cp "/media/backup/m93_cpre_2022-11-12_-_00h00.tgz" "/media/backup-extended-us1/m93_cpre_2022-11-12_-_00h00.tgz" After 1 minute, RAM usage is about 99%. After 2 minutes, VPS take ~1 minute to connect by SSH with new connection. After 4-5 minutes, VPS is no longer responding for 24 hours+, I need to hard restart it. CPU usage is very low, I/O too, bandwidth same... I don't know if this high memory usage is normal, or if I do something wrong, i'm new to s3fs since 2 days. Thank's.

kerem commented

2026-03-04 01:50:53 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 26, 2023):

@Messiah93 I'm sorry for my late reply.
Are you downloading or uploading a 7GB file?
This is my guess but the tmpfs is used, and there is a possibility that a 7GB file is created as a temporary(or cache file) in the tmpfs volume.
In this case, using tmpfs reduces memory usage, so it may match your symptoms.
If so, you can specify the use_cache option to place the cache file on a volume other than tmpfs, or limit disk usage with the ensure_diskfree option.
Please try to use these option.
Thanks in advance for your assistance.

@ggtakec commented on GitHub (Mar 26, 2023): @Messiah93 I'm sorry for my late reply. _Are you downloading or uploading a 7GB file?_ This is my guess but the tmpfs is used, and there is a possibility that a 7GB file is created as a temporary(or cache file) in the tmpfs volume. In this case, using tmpfs reduces memory usage, so it may match your symptoms. If so, you can specify the `use_cache` option to place the cache file on a volume other than tmpfs, or limit disk usage with the `ensure_diskfree` option. Please try to use these option. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:50:53 +03:00

Author

Owner

@Messiah93 commented on GitHub (Mar 29, 2023):

Hi,

No i'm not downloading or uploading a 7gb file... I'm using S3FS for my Plex Media Server... I just try to use the "use_cache" option like you said, and when I use it and start a library scan on Plex, S3FS is litterally downloading all my bucket in local to scan each file, and don't delete these until the scan of all files is not finished in plex process... So, without "use_cache" option, it's fully using my ram and swap... I don't know why it happens, I see a lot of peoples using S3FS with plex without problem...

Thank's for help, now I know where come from the problem, now i have to found why.

@Messiah93 commented on GitHub (Mar 29, 2023): Hi, No i'm not downloading or uploading a 7gb file... I'm using S3FS for my Plex Media Server... I just try to use the "use_cache" option like you said, and when I use it and start a library scan on Plex, S3FS is litterally downloading all my bucket in local to scan each file, and don't delete these until the scan of all files is not finished in plex process... So, without "use_cache" option, it's fully using my ram and swap... I don't know why it happens, I see a lot of peoples using S3FS with plex without problem... Thank's for help, now I know where come from the problem, now i have to found why.

kerem commented

2026-03-04 01:50:54 +03:00

Author

Owner

@ggtakec commented on GitHub (Apr 3, 2023):

@Messiah93
If you use the use_cache option, create a cache file under the specified directory.
If this option is not specified, temporary work files are created and they are created in /tmp(default).
If you want to change this /tmp, you can use tmpdir option.

In either case, Plex Media Server(?) scans for files and seems to use cache or temporary files, so I think you may want to put those files in a non-tmpfs directory.

@ggtakec commented on GitHub (Apr 3, 2023): @Messiah93 If you use the `use_cache` option, create a cache file under the specified directory. If this option is not specified, temporary work files are created and they are created in `/tmp`(default). If you want to change this `/tmp`, you can use `tmpdir` option. In either case, `Plex Media Server(?)` scans for files and seems to use cache or temporary files, so I think you may want to put those files in a non-`tmpfs` directory.