[GH-ISSUE #1016] Trouble Disabling Cache Once Set #558

Open
opened 2026-03-04 01:46:43 +03:00 by kerem · 11 comments
Owner

Originally created by @jamiejackson on GitHub (Apr 18, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1016

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.85(commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Name : fuse
Version : 2.9.2
Release : 11.el7
Architecture: x86_64

Kernel information (uname -r)

3.10.0-957.1.3.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

s3fs command line used, if applicable

/etc/fstab entry, if applicable

s3fs#files-dev.mysite.com:/legacy_media_share/ /mnt/s3/files-dev fuse _netdev,nonempty,iam_role,allow_other,use_path_request_style,use_cache='',url=http://s3.amazonaws.com,uid=root,gid=hudx_admin,umask=0002,rw 0 0
/dev/mapper/rhel-swap swap swap defaults 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

What are the minimum steps needed to go from using cache (I'd had use_cache=/tmp) to disabling it? I tried:

  • Both removing the use_cache option altogether and having use_cache=""
  • sudo umount /mnt/s3/files-dev
  • sudo rm -rf /tmp/files-dev.mysite.com
  • sudo mount /mnt/s3/files-dev

Then I accessed a file in the mount point, and /tmp/files-dev.mysite.com reappeared with the file in it.

I'm having trouble disabling it.

Originally created by @jamiejackson on GitHub (Apr 18, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1016 #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.85(commit:unknown) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Name : fuse Version : 2.9.2 Release : 11.el7 Architecture: x86_64 #### Kernel information (uname -r) 3.10.0-957.1.3.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" #### s3fs command line used, if applicable ``` ``` #### /etc/fstab entry, if applicable ``` s3fs#files-dev.mysite.com:/legacy_media_share/ /mnt/s3/files-dev fuse _netdev,nonempty,iam_role,allow_other,use_path_request_style,use_cache='',url=http://s3.amazonaws.com,uid=root,gid=hudx_admin,umask=0002,rw 0 0 /dev/mapper/rhel-swap swap swap defaults 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue What are the minimum steps needed to go from using cache (I'd had `use_cache=/tmp`) to disabling it? I tried: * Both removing the `use_cache` option altogether and having `use_cache=""` * `sudo umount /mnt/s3/files-dev` * `sudo rm -rf /tmp/files-dev.mysite.com` * `sudo mount /mnt/s3/files-dev` Then I accessed a file in the mount point, and `/tmp/files-dev.mysite.com` reappeared with the file in it. I'm having trouble disabling it.
Author
Owner

@jamiejackson commented on GitHub (Apr 19, 2019):

FYI, a reboot seemed to take care of it, but it would be nice to know the surgical, vs. sledgehammer, approach.

<!-- gh-comment-id:484929349 --> @jamiejackson commented on GitHub (Apr 19, 2019): FYI, a reboot seemed to take care of it, but it would be nice to know the surgical, vs. sledgehammer, approach.
Author
Owner

@ggtakec commented on GitHub (Apr 22, 2019):

@jamiejackson It was easy, but I also tried. However, it could not be reproduced.
(In addition, to disable use_cache, please do not specify use_cache option.)

It doesn't matter, but I think that the following directories exist, so it is recommended to delete them together.
/tmp/.files-dev.mysite.com.mirror
/tmp/.files-dev.mysite.com.stat

But I have not been reproduced it yet, so please let me know if there is any additional information.
Thanks in advance for your help.

<!-- gh-comment-id:485428745 --> @ggtakec commented on GitHub (Apr 22, 2019): @jamiejackson It was easy, but I also tried. However, it could not be reproduced. (In addition, to disable use_cache, please do not specify use_cache option.) It doesn't matter, but I think that the following directories exist, so it is recommended to delete them together. /tmp/.files-dev.mysite.com.mirror /tmp/.files-dev.mysite.com.stat But I have not been reproduced it yet, so please let me know if there is any additional information. Thanks in advance for your help.
Author
Owner

@jamiejackson commented on GitHub (Apr 22, 2019):

Hi @ggtakec, I did try, in turn, both removing the use_cache option and setting it to empty string, but neither seemed to disable it with just a re-mount.

As I mentioned, it did work after rebooting the server, but that's not always practical.

<!-- gh-comment-id:485455145 --> @jamiejackson commented on GitHub (Apr 22, 2019): Hi @ggtakec, I did try, in turn, both removing the `use_cache` option and setting it to empty string, but neither seemed to disable it with just a re-mount. As I mentioned, it did work after rebooting the server, but that's not always practical.
Author
Owner

@rhysmeister commented on GitHub (Apr 24, 2019):

I'm having similar issues...

fuse 2.9.3
s3fs 1.85
Redhat Ent 6.7

If I set use_cache="" it creates a directory on / called "" (yep double quotes) and uses that as cache. I have to remove it with rm -Rf \"\" (that makes me nervous). I've also messed around with the ensure_diskfree variable but this doesn't seem to help (it just bails when it hits the limit). Rebooting doesn't seem to help.

My mounts are set as follows...

s3fs#XXXXXXXX /mnt/XXXXXXXX fuse url=https://customendpoint.com,_netdev,allow_other,max_stat_cache_size=0,ensure_diskfree=500,multipart_size=10 0 0

Is there a way to force multipart upload?

<!-- gh-comment-id:486203274 --> @rhysmeister commented on GitHub (Apr 24, 2019): I'm having similar issues... fuse 2.9.3 s3fs 1.85 Redhat Ent 6.7 If I set use_cache="" it creates a directory on / called "" (yep double quotes) and uses that as cache. I have to remove it with rm -Rf \\"\\" (that makes me nervous). I've also messed around with the ensure_diskfree variable but this doesn't seem to help (it just bails when it hits the limit). Rebooting doesn't seem to help. My mounts are set as follows... `s3fs#XXXXXXXX /mnt/XXXXXXXX fuse url=https://customendpoint.com,_netdev,allow_other,max_stat_cache_size=0,ensure_diskfree=500,multipart_size=10 0 0` Is there a way to force multipart upload?
Author
Owner

@jamiejackson commented on GitHub (May 2, 2019):

@rhysmeister , I just came across the behavior you described. The cache is tenacious!

FWIW, I had use_cache='', so I ended up with a cache directory of /''/<mybucket> :-/

Now I'm back the situation where I can't get rid of the cache (without rebooting), because one affected server has used use_cache='' during this boot, and I'll probably have to remove the option and reboot to get rid of the caching.

@rhysmeister , the only way I've been able to stop the cache is to remove the use_cache option completely then reboot. Have you done exactly that yet?

<!-- gh-comment-id:488713038 --> @jamiejackson commented on GitHub (May 2, 2019): @rhysmeister , I just came across the behavior you described. The cache is tenacious! FWIW, I had `use_cache=''`, so I ended up with a cache directory of `/''/<mybucket>` :-/ Now I'm back the situation where I can't get rid of the cache (without rebooting), because one affected server has used `use_cache=''` during this boot, and I'll probably have to remove the option and reboot to get rid of the caching. @rhysmeister , the only way I've been able to stop the cache is to _remove_ the `use_cache` option completely then reboot. Have you done exactly that yet?
Author
Owner

@jamiejackson commented on GitHub (May 2, 2019):

Maintainers: In addition to whatever investigation you might be doing, could you please modify the documentation to state that one must remove the option completely to disable the cache? Otherwise, other folks will continue to get bitten in the interim.

<!-- gh-comment-id:488715639 --> @jamiejackson commented on GitHub (May 2, 2019): Maintainers: In addition to whatever investigation you might be doing, could you please modify the documentation to state that one _must_ remove the option completely to disable the cache? Otherwise, other folks will continue to get bitten in the interim.
Author
Owner

@jamiejackson commented on GitHub (May 2, 2019):

@ggtakec: FYI, I might be able to get around to enabling those debug settings and doing some testing at some point, but I won't be able to get to it right away.

<!-- gh-comment-id:488716106 --> @jamiejackson commented on GitHub (May 2, 2019): @ggtakec: FYI, I might be able to get around to enabling those debug settings and doing some testing at some point, but I won't be able to get to it right away.
Author
Owner

@ggtakec commented on GitHub (May 4, 2019):

@jamiejackson
Did this problem occur when "" or '' is given to use_cache option?

I was able to confirm the same symptom as this problem by passing use_cache=\"\".
(In practice, normally(not root user) cache files could not be created directly under / due to permission issues)

Could you not pass double_quotes or single quotes for use_cache?
(If you run s3fs from shell, It is necessary to escape quotes.)

I think that it is possible to strengthen the check of options of s3fs, but there is also the issue of how much to check.

<!-- gh-comment-id:489341176 --> @ggtakec commented on GitHub (May 4, 2019): @jamiejackson Did this problem occur when `""` or `''` is given to use_cache option? I was able to confirm the same symptom as this problem by passing `use_cache=\"\"`. (In practice, normally(not root user) cache files could not be created directly under `/` due to permission issues) Could you not pass double_quotes or single quotes for use_cache? (If you run s3fs from shell, It is necessary to escape quotes.) I think that it is possible to strengthen the check of options of s3fs, but there is also the issue of how much to check.
Author
Owner

@jamiejackson commented on GitHub (May 6, 2019):

Did this problem occur when "" or '' is given to use_cache option?

@rhysmeister (with "") and I (with '') both had the problem of s3fs creating a directory in the root named as quotes.

Could you not pass double_quotes or single quotes for use_cache?

There's nothing keeping us from doing so, but it causes the problem we've already described. (It doesn't disable cache as the documentation claims it should.)

<!-- gh-comment-id:489589519 --> @jamiejackson commented on GitHub (May 6, 2019): > Did this problem occur when "" or '' is given to use_cache option? @rhysmeister (with `""`) and I (with `''`) both had the problem of s3fs creating a directory in the root named as quotes. > Could you not pass double_quotes or single quotes for use_cache? There's nothing keeping us from doing so, but it causes the problem we've already described. (It doesn't disable cache as the documentation claims it should.)
Author
Owner

@rhysmeister commented on GitHub (May 6, 2019):

@jamiejackson I couldn't turn off the cache. It insisted on using /tmp. I threw some disk space at it.

<!-- gh-comment-id:489653596 --> @rhysmeister commented on GitHub (May 6, 2019): @jamiejackson I couldn't turn off the cache. It insisted on using /tmp. I threw some disk space at it.
Author
Owner

@zjtheone commented on GitHub (May 26, 2019):

I'm having similar issues...,Has anyone solved this issue? Thanks

<!-- gh-comment-id:495982943 --> @zjtheone commented on GitHub (May 26, 2019): I'm having similar issues...,Has anyone solved this issue? Thanks
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#558
No description provided.