kerem commented

Owner

Originally created by @jpeacock-mypoints on GitHub (Sep 16, 2015).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/269

My local filesystem fills up when I'm transferring files. Any files transferred to S3 are truncated if their size is greater than the amount of free space locally available. During the most recent test I used only one file and found that the amount of filesystem space used before and during the transfer increases by the file size. I've used two version of the s3fs mount command (one with and one without the use_cache options) but neither affect the issue. I've run DU to try to isolate a temp directory or scratch file, but /home is the same size no matter when I check (3.9GB).

Below is an example of what happens to a small filesystem when I run:
"mv /home/jpeacock/rhel* /backups. I removed most of the lines for brevity, but I think you'll see the problem.

Thanks in advance!

--Jonathan

Mount

s3fs dev2.mpb /backups/ -o passwd_file=/root/.s3fs -o use_sse
s3fs dev2.mpb /backups/ -o passwd_file=/root/.s3fs -o use_sse -o use_cache=""

DU

[root@dev5 jpeacock]# du -hs /*
26K     /backups
0       /bin
95M     /boot
0       /dev
20M     /etc
3.9G    /home
0       /lib
0       /lib64
0       /media
0       /mnt
0       /opt

while loop

[root@dev5 jpeacock]# while true; do df -h / ;ls -alh /backups/rhel*;ls -alh /home/jpeacock/rhel*;echo =====;sleep 60;done
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/centos_dev5-root   14G  4.7G  9.3G  34% /
ls: cannot access /backups/rhel*: No such file or directory
-rw-r--r--. 1 root root 3.6G Sep 16 13:49 /home/jpeacock/rhel-workstation-6.4-i386-dvd.iso
=====
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/centos_dev5-root   14G  8.7G  5.3G  63% /
-rw-------. 1 root root 0 Sep 16 13:49 /backups/rhel-workstation-6.4-i386-dvd.iso
-rw-r--r--. 1 root root 3.6G Sep 16 13:49 /home/jpeacock/rhel-workstation-6.4-i386-dvd.iso
=====
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/centos_dev5-root   14G  8.2G  5.8G  59% /
-rw-------. 1 root root 0 Sep 16 13:49 /backups/rhel-workstation-6.4-i386-dvd.iso
-rw-r--r--. 1 root root 3.6G Sep 16 13:49 /home/jpeacock/rhel-workstation-6.4-i386-dvd.iso
=====
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/centos_dev5-root   14G  1.1G   13G   8% /
-rw-------. 1 root root 3.6G Sep 16 14:01 /backups/rhel-workstation-6.4-i386-dvd.iso
ls: cannot access /home/jpeacock/rhel*: No such file or directory
=====

Version

[root@dev5 ~]# s3fs --version
Amazon Simple Storage Service File System V1.79 with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally created by @jpeacock-mypoints on GitHub (Sep 16, 2015). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/269 My local filesystem fills up when I'm transferring files. Any files transferred to S3 are truncated if their size is greater than the amount of free space locally available. During the most recent test I used only one file and found that the amount of filesystem space used before and during the transfer increases by the file size. I've used two version of the s3fs mount command (one with and one without the use_cache options) but neither affect the issue. I've run DU to try to isolate a temp directory or scratch file, but /home is the same size no matter when I check (3.9GB). Below is an example of what happens to a small filesystem when I run: "mv /home/jpeacock/rhel\* /backups. I removed most of the lines for brevity, but I think you'll see the problem. Thanks in advance! --Jonathan # Mount ``` s3fs dev2.mpb /backups/ -o passwd_file=/root/.s3fs -o use_sse s3fs dev2.mpb /backups/ -o passwd_file=/root/.s3fs -o use_sse -o use_cache="" ``` # DU ``` [root@dev5 jpeacock]# du -hs /* 26K /backups 0 /bin 95M /boot 0 /dev 20M /etc 3.9G /home 0 /lib 0 /lib64 0 /media 0 /mnt 0 /opt ``` # while loop ``` [root@dev5 jpeacock]# while true; do df -h / ;ls -alh /backups/rhel*;ls -alh /home/jpeacock/rhel*;echo =====;sleep 60;done Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_dev5-root 14G 4.7G 9.3G 34% / ls: cannot access /backups/rhel*: No such file or directory -rw-r--r--. 1 root root 3.6G Sep 16 13:49 /home/jpeacock/rhel-workstation-6.4-i386-dvd.iso ===== Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_dev5-root 14G 8.7G 5.3G 63% / -rw-------. 1 root root 0 Sep 16 13:49 /backups/rhel-workstation-6.4-i386-dvd.iso -rw-r--r--. 1 root root 3.6G Sep 16 13:49 /home/jpeacock/rhel-workstation-6.4-i386-dvd.iso ===== Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_dev5-root 14G 8.2G 5.8G 59% / -rw-------. 1 root root 0 Sep 16 13:49 /backups/rhel-workstation-6.4-i386-dvd.iso -rw-r--r--. 1 root root 3.6G Sep 16 13:49 /home/jpeacock/rhel-workstation-6.4-i386-dvd.iso ===== Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_dev5-root 14G 1.1G 13G 8% / -rw-------. 1 root root 3.6G Sep 16 14:01 /backups/rhel-workstation-6.4-i386-dvd.iso ls: cannot access /home/jpeacock/rhel*: No such file or directory ===== ``` # Version ``` [root@dev5 ~]# s3fs --version Amazon Simple Storage Service File System V1.79 with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ```

kerem closed this issue

2026-03-04 01:42:31 +03:00

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@gojomo commented on GitHub (Sep 23, 2015):

I have a variant of the same problem. I have a large (70GB) file in S3, but only ~20GB free space on my machine's local volume (which happens to be AWS EBS).

My process would like to read various ranges from the large file (eventually the whole file), without mirroring the entire contents locally, not even temporarily (such as for the open duration of the process's file handles) – because of course the full 70GB won't fit. I'm already running with use_cache="".

However, currently all local space gets consumed by s3fs – apparently via temporary cache files under /tmp that aren't part of the usual caching mechanism. (I believe s3fs is creating these then deleting them with a retained open handle, so that they'll only live as long as s3fs holds that open handle. But the net effect is still consuming all local space in a futile attempt to retain all of the remote file locally.)

An ideal fix would be a configurable cap on how much disk s3fs uses in this manner, and/or an option to never use any local disk in this manner. (Especially in my case, where the 'local' disk is actually EBS, there's likely no net performance/budget gain from reducing S3 ops at the cost of just as many EBS ops. I'd even rather have s3fs use a RAM disk for tmp-storage; this machine has more RAM than EBS volume space.)

A quick workaround might be to recycle these phantom cache files more rapidly, perhaps after a certain amount of time and/or data is involved, so that the total local consumption remains a capped function of the number of still-open user-process handles, rather than a function of the total data pushed through a single user-process handle.

(With a pointer to the code sections where this non-"cache" local-disk-usage is controlled, I might be able to offer a patch/workaround...)

@gojomo commented on GitHub (Sep 23, 2015): I have a variant of the same problem. I have a large (70GB) file in S3, but only ~20GB free space on my machine's local volume (which happens to be AWS EBS). My process would like to read various ranges from the large file (eventually the whole file), without mirroring the entire contents locally, not even temporarily (such as for the open duration of the process's file handles) – because of course the full 70GB won't fit. I'm already running with `use_cache=""`. However, currently all local space gets consumed by s3fs – apparently via temporary cache files under /tmp that aren't part of the usual caching mechanism. (I believe s3fs is creating these then deleting them with a retained open handle, so that they'll only live as long as s3fs holds that open handle. But the net effect is still consuming all local space in a futile attempt to retain all of the remote file locally.) An ideal fix would be a configurable cap on how much disk s3fs uses in this manner, and/or an option to _never_ use any local disk in this manner. (Especially in my case, where the 'local' disk is actually EBS, there's likely no net performance/budget gain from reducing S3 ops at the cost of just as many EBS ops. I'd even rather have s3fs use a RAM disk for tmp-storage; this machine has more RAM than EBS volume space.) A quick workaround might be to recycle these phantom cache files more rapidly, perhaps after a certain amount of time and/or data is involved, so that the total local consumption remains a capped function of the number of still-open user-process handles, rather than a function of the total data pushed through a single user-process handle. (With a pointer to the code sections where this non-"cache" local-disk-usage is controlled, I might be able to offer a patch/workaround...)

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@gojomo commented on GitHub (Sep 29, 2015):

Hi, @ggtakec – any chance of a pointer to the code which creates these phantom /tmp/* files?

@gojomo commented on GitHub (Sep 29, 2015): Hi, @ggtakec – any chance of a pointer to the code which creates these phantom /tmp/\* files?

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 30, 2015):

@gojomo I remember there is a similar problem before.
And then we did not modify the codes about this problem at that time.
Now I will try to deal with this problem, because s3fs could not upload huge file if system does not have free disk space for it.

But this code change takes a lot of time for me, please wait a while.

Thanks in advance.

@ggtakec commented on GitHub (Sep 30, 2015): @gojomo I remember there is a similar problem before. And then we did not modify the codes about this problem at that time. Now I will try to deal with this problem, because s3fs could not upload huge file if system does not have free disk space for it. But this code change takes a lot of time for me, please wait a while. Thanks in advance.

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@gojomo commented on GitHub (Oct 6, 2015):

Thanks for your attention. Let me know if there's a way I can help code/test/etc!

@gojomo commented on GitHub (Oct 6, 2015): Thanks for your attention. Let me know if there's a way I can help code/test/etc!

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@deguich commented on GitHub (Oct 9, 2015):

Hi, same problem here when reading big files from s3fs. Local filesystem is growing until full disk. That produce a reading error in the client application.

Another subject, it's an Idea for future release :
I use s3fs with a ceph cluster for backup with a tsm server. I try to update code for adding a root level at mount point. "Root level" goal is to list all buckets using the Ceph admin ops API. I use this request : "GET admin/metadata/bucket" to get all the bucket name.

A CephAdmin option to mount all the "Ceph Object Gateway" (the S3 like sever for ceph) like I try to do could be a good improvement for all the users who want to backup their ceph cluster !

@deguich commented on GitHub (Oct 9, 2015): Hi, same problem here when reading big files from s3fs. Local filesystem is growing until full disk. That produce a reading error in the client application. Another subject, it's an Idea for future release : I use s3fs with a ceph cluster for backup with a tsm server. I try to update code for adding a root level at mount point. "Root level" goal is to list all buckets using the Ceph admin ops API. I use this request : "GET admin/metadata/bucket" to get all the bucket name. A CephAdmin option to mount all the "Ceph Object Gateway" (the S3 like sever for ceph) like I try to do could be a good improvement for all the users who want to backup their ceph cluster !

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@deguich commented on GitHub (Oct 9, 2015):

I tried to use a ramdisk on the use_cache path, unmount /mnt/exports3fs don't free the memory.
bucket dir and .bucket.stat dir are still in ramdisk.

@deguich commented on GitHub (Oct 9, 2015): I tried to use a ramdisk on the use_cache path, unmount /mnt/exports3fs don't free the memory. bucket dir and .bucket.stat dir are still in ramdisk.

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@ggtakec commented on GitHub (Oct 20, 2015):

@gojomo and @deguich
I updated master branch by #280 PR.(Please see it for detail)
It supported a object which is larger than free disk space which is the disk for use_cache option path or temporary file on system.
You can specify the free disk space size which you need to keep for free space by ensure_diskfree new option.
If you do not specify it, s3fs sets default value for it as same as multipart_size.

Please try to use latest codes in master branch.
Thanks in advance for your help.

@ggtakec commented on GitHub (Oct 20, 2015): @gojomo and @deguich I updated master branch by #280 PR.(Please see it for detail) It supported a object which is larger than free disk space which is the disk for use_cache option path or temporary file on system. You can specify the free disk space size which you need to keep for free space by ensure_diskfree new option. If you do not specify it, s3fs sets default value for it as same as multipart_size. Please try to use latest codes in master branch. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 17, 2016):

I'm closing this issue, if you have a problem yet, please post new issue or reopen this issue.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 17, 2016): I'm closing this issue, if you have a problem yet, please post new issue or reopen this issue. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@dellelce commented on GitHub (Sep 29, 2017):

I am having a similar issue but files are kept open even as deleted:

this is the output from lsof:

s3fs       2584     root  588u      REG              202,1     7834316         85 /tmp/tmpfwkzkYz (deleted)
s3fs       2584     root  589u      REG              202,1       24565         77 /tmp/tmpfTZY7oC (deleted)
s3fs       2584     root  590u      REG              202,1    44667972         87 /tmp/tmpf7GI8bY (deleted)
s3fs       2584     root  591u      REG              202,1    75719392         88 /tmp/tmpfhQs00w (deleted)
s3fs       2584     root  592u      REG              202,1   134380296        105 /tmp/tmpfFbOKiJ (deleted)
s3fs       2584     root  593u      REG              202,1       24727        257 /tmp/tmpfCVQ5Fk (deleted)

used version is 1.80.

@dellelce commented on GitHub (Sep 29, 2017): I am having a similar issue but files are kept open even as deleted: this is the output from lsof: ``` s3fs 2584 root 588u REG 202,1 7834316 85 /tmp/tmpfwkzkYz (deleted) s3fs 2584 root 589u REG 202,1 24565 77 /tmp/tmpfTZY7oC (deleted) s3fs 2584 root 590u REG 202,1 44667972 87 /tmp/tmpf7GI8bY (deleted) s3fs 2584 root 591u REG 202,1 75719392 88 /tmp/tmpfhQs00w (deleted) s3fs 2584 root 592u REG 202,1 134380296 105 /tmp/tmpfFbOKiJ (deleted) s3fs 2584 root 593u REG 202,1 24727 257 /tmp/tmpfCVQ5Fk (deleted) ``` used version is 1.80.

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@gaul commented on GitHub (Oct 2, 2017):

@dellelce Please test with the latest master branch and open a new issue if these symptoms persist. Also share the exact command-line flags you use and which operations reproduce the symptoms.

@gaul commented on GitHub (Oct 2, 2017): @dellelce Please test with the latest master branch and open a new issue if these symptoms persist. Also share the exact command-line flags you use and which operations reproduce the symptoms.

kerem commented

2026-03-04 01:42:32 +03:00

Author

Owner

@raj-aws commented on GitHub (Dec 7, 2020):

Hi, I am facing the simialr issue, I have created a mount like below
s3fs#s3bucket /s3mnt fuse passwd_file=/root/.passwd-s3fs,use_cache=/root/s3fs-cache,allow_other,umask=0022,gid=3001,uid=5001 0 0

The /root/s3fs-cache filesystem is quickly filling up, can you please help how can i stop S3fs to clear the filesystem regularly or not use cache

Please provide best practice of creating a mount.

Thanks for the help

Regards
Raj

@raj-aws commented on GitHub (Dec 7, 2020): Hi, I am facing the simialr issue, I have created a mount like below s3fs#s3bucket /s3mnt fuse passwd_file=/root/.passwd-s3fs,use_cache=/root/s3fs-cache,allow_other,umask=0022,gid=3001,uid=5001 0 0 The /root/s3fs-cache filesystem is quickly filling up, can you please help how can i stop S3fs to clear the filesystem regularly or not use cache Please provide best practice of creating a mount. Thanks for the help Regards Raj

kerem referenced this issue

2026-03-04 01:53:33 +03:00

[PR #136] [MERGED] Small fixes to integration tests #1359

Rows
Columns

[GH-ISSUE #269] Local filesystem filling up #136

Mount

DU

while loop

Version