[GH-ISSUE #2694] Cache fills up whole fs on write when tmpfs is not mounted on /tmp #1281

New issue

Open

opened 2026-03-04 01:52:45 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 01:52:45 +03:00

Owner

Originally created by @grebokra on GitHub (Jul 25, 2025).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2694

Additional Information

Version of s3fs being used (`s3fs --version`)

# s3fs --version
Amazon Simple Storage Service File System V1.95 with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

# rpm -qi s3fs-fuse
Name        : s3fs-fuse
Version     : 1.95
Release     : 1.el8
Architecture: x86_64
Install Date: Wed 16 Jul 2025 10:16:15 AM MSK
Group       : Unspecified
Size        : 922972
License     : GPL-2.0-or-later
Signature   : RSA/SHA256, Sun 27 Oct 2024 09:37:38 PM MSK, Key ID 21ea45ab2f86d6a1
Source RPM  : s3fs-fuse-1.95-1.el8.src.rpm
Build Date  : Sun 27 Oct 2024 09:32:00 PM MSK
Build Host  : buildvm-x86-04.iad2.fedoraproject.org
Relocations : (not relocatable)
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://github.com/s3fs-fuse/s3fs-fuse
Bug URL     : https://bugz.fedoraproject.org/s3fs-fuse
Summary     : FUSE-based file system backed by Amazon S3
Description :
s3fs is a FUSE file system that allows you to mount an Amazon S3 bucket as a
local file system. It stores files natively and transparently in S3 (i.e.,
you can use other programs to access the same files). Maximum file size is
5 TB when using multipart upload.

s3fs is stable and is being used in number of production environments, e.g.,
rsync backup to s3.

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

# rpm -qi fuse
Name        : fuse
Version     : 2.9.7
Release     : 16.el8
Architecture: x86_64
Install Date: Wed 16 Jul 2025 10:16:15 AM MSK
Group       : Unspecified
Size        : 208300
License     : GPL+
Signature   : RSA/SHA256, Mon 10 Oct 2022 01:51:47 PM MSK, Key ID 51d6647ec21ad6ea
Source RPM  : fuse-2.9.7-16.el8.src.rpm
Build Date  : Sat 08 Oct 2022 03:38:05 PM MSK
Build Host  : x64-builder02.almalinux.org
Relocations : (not relocatable)
Packager    : AlmaLinux Packaging Team <packager@almalinux.org>
Vendor      : AlmaLinux
URL         : http://fuse.sf.net
Summary     : File System in Userspace (FUSE) v2 utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE v2 userspace tools to
mount a FUSE filesystem.

Provider (`AWS`, `OVH`, `Hetzner`, `iDrive E2`, ...)

Ceph RadosGW

Kernel information (`uname -r`)

4.18.0-477.10.1.el8_8.x86_64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

# cat /etc/redhat-release
AlmaLinux release 8.8 (Sapphire Caracal)

How to run s3fs, if applicable

# cat /etc/fstab
...
asdasd  /backups        fuse.s3fs       _netdev,allow_other,dbglevel=info,nomixupload,multipart_size=256,use_path_request_style,url=https://s3.mycloud.cloud.internal:443,passwd_file=/etc/passwd-s3fs  0       0

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

N/A

Details about issue

I've noticed a strange behaviour when I upload big files (about ~512GiB in size) to s3 bucket via s3fs-fuse.

Here is an example of how files are being written:

spit_some_data_to_stdout | gzip -c > /backups/my_data_backup.gz

The man page for s3fs states the following:

          Without local cache

       Since s3fs always requires some storage space for operation, it creates temporary files to store incoming  write  requests  until  the
       required  s3  request  size  is reached and the segment has been uploaded. After that, this data is truncated in the temporary file to
       free up storage space.

       Per file you need at least twice the part size (default 5MB or "-o multipart_size") for writing multipart requests or  space  for  the
       whole file if single requests are enabled ("-o nomultipart").

Therefore, I assume that s3fs should use as much space in the filesystem for /tmp directory as it takes for 2 parts (in my case it is ~512MiB) of multipart upload at a given moment in time.

When tmpfs is mounted in /tmp not much space is being used. There is a cache file in tmp that I can see using lsof, and the number in the SIZE column increases during writing, but tmpfs does not fill up as I see in df -h, so I assume everything works as expected (and the file write finishes without an issue after whole 512G file is uploaded to the bucket)

The behavior seems to be different when tmpfs is not mounted in /tmp, and xfs (rootfs) is used for storing cache data. The filesystem fills up quickly.

Here is the output of xfs_info, if it can help in any way

# xfs_info /
meta-data=/dev/vda1              isize=512    agcount=17, agsize=512000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=8388347, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

I assume there might be some kind of bug regarding handling truncation of cache file.

If I'm doing something wrong, please let me know :^)

P.S.: I use nomixupload option, because ceph rgw handles copypart operations very slowly... therefore one multipart upload without copying is much better in my case.

Originally created by @grebokra on GitHub (Jul 25, 2025). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2694  ### Additional Information #### Version of s3fs being used (`s3fs --version`) ``` # s3fs --version Amazon Simple Storage Service File System V1.95 with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # rpm -qi s3fs-fuse Name : s3fs-fuse Version : 1.95 Release : 1.el8 Architecture: x86_64 Install Date: Wed 16 Jul 2025 10:16:15 AM MSK Group : Unspecified Size : 922972 License : GPL-2.0-or-later Signature : RSA/SHA256, Sun 27 Oct 2024 09:37:38 PM MSK, Key ID 21ea45ab2f86d6a1 Source RPM : s3fs-fuse-1.95-1.el8.src.rpm Build Date : Sun 27 Oct 2024 09:32:00 PM MSK Build Host : buildvm-x86-04.iad2.fedoraproject.org Relocations : (not relocatable) Packager : Fedora Project Vendor : Fedora Project URL : https://github.com/s3fs-fuse/s3fs-fuse Bug URL : https://bugz.fedoraproject.org/s3fs-fuse Summary : FUSE-based file system backed by Amazon S3 Description : s3fs is a FUSE file system that allows you to mount an Amazon S3 bucket as a local file system. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files). Maximum file size is 5 TB when using multipart upload. s3fs is stable and is being used in number of production environments, e.g., rsync backup to s3. ``` #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)  ``` # rpm -qi fuse Name : fuse Version : 2.9.7 Release : 16.el8 Architecture: x86_64 Install Date: Wed 16 Jul 2025 10:16:15 AM MSK Group : Unspecified Size : 208300 License : GPL+ Signature : RSA/SHA256, Mon 10 Oct 2022 01:51:47 PM MSK, Key ID 51d6647ec21ad6ea Source RPM : fuse-2.9.7-16.el8.src.rpm Build Date : Sat 08 Oct 2022 03:38:05 PM MSK Build Host : x64-builder02.almalinux.org Relocations : (not relocatable) Packager : AlmaLinux Packaging Team <packager@almalinux.org> Vendor : AlmaLinux URL : http://fuse.sf.net Summary : File System in Userspace (FUSE) v2 utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE v2 userspace tools to mount a FUSE filesystem. ``` #### Provider (`AWS`, `OVH`, `Hetzner`, `iDrive E2`, ...)  - Ceph RadosGW #### Kernel information (`uname -r`)  ```# uname -r 4.18.0-477.10.1.el8_8.x86_64 ``` #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`)  ``` # cat /etc/redhat-release AlmaLinux release 8.8 (Sapphire Caracal) ``` #### How to run s3fs, if applicable ``` # cat /etc/fstab ... asdasd /backups fuse.s3fs _netdev,allow_other,dbglevel=info,nomixupload,multipart_size=256,use_path_request_style,url=https://s3.mycloud.cloud.internal:443,passwd_file=/etc/passwd-s3fs 0 0 ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  N/A ### Details about issue I've noticed a strange behaviour when I upload big files (about ~512GiB in size) to s3 bucket via s3fs-fuse. Here is an example of how files are being written: ``` spit_some_data_to_stdout | gzip -c > /backups/my_data_backup.gz ``` The man page for s3fs states the following: ``` Without local cache Since s3fs always requires some storage space for operation, it creates temporary files to store incoming write requests until the required s3 request size is reached and the segment has been uploaded. After that, this data is truncated in the temporary file to free up storage space. Per file you need at least twice the part size (default 5MB or "-o multipart_size") for writing multipart requests or space for the whole file if single requests are enabled ("-o nomultipart"). ``` Therefore, I assume that s3fs should use as much space in the filesystem for /tmp directory as it takes for 2 parts (in my case it is ~512MiB) of multipart upload at a given moment in time. When `tmpfs` is mounted in /tmp not much space is being used. There is a cache file in tmp that I can see using `lsof`, and the number in the SIZE column increases during writing, but `tmpfs` does not fill up as I see in `df -h`, so I assume everything works as expected (and the file write finishes without an issue after whole 512G file is uploaded to the bucket) The behavior seems to be different when `tmpfs` is not mounted in /tmp, and `xfs` (rootfs) is used for storing cache data. The filesystem fills up quickly. Here is the output of `xfs_info`, if it can help in any way ``` # xfs_info / meta-data=/dev/vda1 isize=512 agcount=17, agsize=512000 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 bigtime=0 inobtcount=0 data = bsize=4096 blocks=8388347, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 ``` I assume there might be some kind of bug regarding handling truncation of cache file. If I'm doing something wrong, please let me know :^) P.S.: I use `nomixupload` option, because ceph rgw handles copypart operations very slowly... therefore one multipart upload without copying is much better in my case.