[GH-ISSUE #1975] Deleted files not removed from cache #998

New issue

Open

opened 2026-03-04 01:50:31 +03:00 by kerem · 4 comments

kerem commented

2026-03-04 01:50:31 +03:00

Owner

Originally created by @maron546 on GitHub (Jul 5, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1975

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Kernel information (uname -r)

4.18.0-348.23.1.el8_5.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"

/etc/fstab entry, if applicable

<bucket> /shared fuse.s3fs _netdev,allow_other,use_path_request_style,url=https://<s3-service>,use_cache=/tmp/s3fs-cache,enable_noobj_cache,uid=<uid>,gid=<gid>,mp_umask=0007 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

/var/log/messages:Jul  5 16:52:59 <server> s3fs[45393]: s3fs version 1.91(unknown) : s3fs -o rw,allow_other,use_path_request_style,url=https://<s3-service>,use_cache=/tmp/s3fs-cache,enable_noobj_cache,uid=<uid>,gid=<gid>,mp_umask=0007,dev,suid <bucket> /shared
/var/log/messages:Jul  5 16:52:59 <server> s3fs[45393]: s3fs version 1.91(unknown) : s3fs -o rw,allow_other,use_path_request_style,url=https://<s3-service>,use_cache=/tmp/s3fs-cache,enable_noobj_cache,uid=<uid>,gid=<gid>,mp_umask=0007,dev,suid <bucket> /shared
/var/log/messages:Jul  5 16:52:59 <server> s3fs[45393]: Loaded mime information from /etc/mime.types
/var/log/messages:Jul  5 16:52:59 <server> s3fs[45393]: Loaded mime information from /etc/mime.types
/var/log/messages:Jul  5 16:53:00 <server> s3fs[45394]: init v1.91(commit:unknown) with OpenSSL
/var/log/messages:Jul  5 16:53:00 <server> s3fs[45394]: init v1.91(commit:unknown) with OpenSSL

Details about issue

We have two nodes that share the same bucket througth s3fs.
Steps to reproduce the issue:

Create a file inside the mountpoint (/shared).
The file is correctly uploaded on the bucket and loaded in local cache (/tmp/s3fs-cache) if opened.
Remove the file from the mountpoint.
The file is deleted from the local cache only on the node on which the command for removing the file has been executed.
The file is still present in the cache of the other node.

Originally created by @maron546 on GitHub (Jul 5, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1975 ### Additional Information #### Version of s3fs being used (s3fs --version) ``` Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Kernel information (uname -r) 4.18.0-348.23.1.el8_5.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` NAME="Red Hat Enterprise Linux" VERSION="8.5 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.5" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL="https://www.redhat.com/" DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.5 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.5" ``` #### /etc/fstab entry, if applicable ``` <bucket> /shared fuse.s3fs _netdev,allow_other,use_path_request_style,url=https://<s3-service>,use_cache=/tmp/s3fs-cache,enable_noobj_cache,uid=<uid>,gid=<gid>,mp_umask=0007 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` /var/log/messages:Jul 5 16:52:59 <server> s3fs[45393]: s3fs version 1.91(unknown) : s3fs -o rw,allow_other,use_path_request_style,url=https://<s3-service>,use_cache=/tmp/s3fs-cache,enable_noobj_cache,uid=<uid>,gid=<gid>,mp_umask=0007,dev,suid <bucket> /shared /var/log/messages:Jul 5 16:52:59 <server> s3fs[45393]: s3fs version 1.91(unknown) : s3fs -o rw,allow_other,use_path_request_style,url=https://<s3-service>,use_cache=/tmp/s3fs-cache,enable_noobj_cache,uid=<uid>,gid=<gid>,mp_umask=0007,dev,suid <bucket> /shared /var/log/messages:Jul 5 16:52:59 <server> s3fs[45393]: Loaded mime information from /etc/mime.types /var/log/messages:Jul 5 16:52:59 <server> s3fs[45393]: Loaded mime information from /etc/mime.types /var/log/messages:Jul 5 16:53:00 <server> s3fs[45394]: init v1.91(commit:unknown) with OpenSSL /var/log/messages:Jul 5 16:53:00 <server> s3fs[45394]: init v1.91(commit:unknown) with OpenSSL ``` ### Details about issue We have two nodes that share the same bucket througth s3fs. Steps to reproduce the issue: 1. Create a file inside the mountpoint (**/shared**). 2. The file is correctly uploaded on the bucket and loaded in local cache (**/tmp/s3fs-cache**) if opened. 3. Remove the file from the mountpoint. 4. The file is deleted from the local cache only on the node on which the command for removing the file has been executed. 5. The file is still present in the cache of the other node.

kerem commented

2026-03-04 01:50:32 +03:00

Author

Owner

@maron546 commented on GitHub (Jul 20, 2022):

Any news?

@maron546 commented on GitHub (Jul 20, 2022): Any news?

kerem commented

2026-03-04 01:50:32 +03:00

Author

Owner

@tb3088 commented on GitHub (Jul 20, 2022):

The file is still present in the cache of the other node.

S3fs is not a multi-node cache-coherent filesystem. S3fs also doesn't detect edits done to the S3 bucket directly either. Right from the documentation.

no coordination between multiple clients mounting the same bucket
inotify detects only local modifications, not external ones by other clients or tools

The only implementation that works correctly is AWS StorageGateway acting as a NFS server.

@tb3088 commented on GitHub (Jul 20, 2022): > The file is still present in the cache of the other node. S3fs is not a multi-node cache-coherent filesystem. S3fs also doesn't detect edits done to the S3 bucket directly either. Right from the documentation. - no coordination between multiple clients mounting the same bucket - inotify detects only local modifications, not external ones by other clients or tools The only implementation that works correctly is AWS StorageGateway acting as a NFS server.

kerem commented

2026-03-04 01:50:32 +03:00

Author

Owner

@hereTac commented on GitHub (May 18, 2023):

In my case, I use s3fs-fuse to mount MinIO S3 with its own configuration for data lifecycle management, where data is recycled after 3 days. Unfortunately, the local cache is unable to detect when data has been deleted by MinIO itself.

@hereTac commented on GitHub (May 18, 2023): In my case, I use s3fs-fuse to mount MinIO S3 with its own configuration for data lifecycle management, where data is recycled after 3 days. Unfortunately, the local cache is unable to detect when data has been deleted by MinIO itself.

kerem commented

2026-03-04 01:50:33 +03:00

Author

Owner

@ggtakec commented on GitHub (May 20, 2023):

s3fs has two types of cache: file object content cache and file object stat information cache.
I think that the reason why the deleted file in another node in this case continues to exist is because the stat cache remains.
Try setting stat_cache_expire, stat_cache_interval_expire, disable_noobj_cache or max_stat_cache_size options like cache out stat cache.

@ggtakec commented on GitHub (May 20, 2023): s3fs has two types of cache: `file object content` cache and `file object stat information` cache. I think that the reason why the deleted file in another node in this case continues to exist is because the `stat cache` remains. Try setting `stat_cache_expire`, `stat_cache_interval_expire`, `disable_noobj_cache` or `max_stat_cache_size` options like cache out stat cache.