[GH-ISSUE #1607] S3fs calls S3 aggressively 1k calls / minute #846

Closed
opened 2026-03-04 01:49:18 +03:00 by kerem · 3 comments
Owner

Originally created by @JeremieDoctrine on GitHub (Mar 18, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1607

Version of s3fs being used (s3fs --version)

1.89

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2

Kernel information (uname -r)

4.14.88-88.76.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

/etc/fstab entry, if applicable

s3fs#elasticsearch-mounted-volume:/dev/ /etc/elasticsearch/shared fuse _netdev,allow_other,use_cache=/tmp/s3fs,iam_role=auto,url=https://s3.amazonaws.com,umask=0007,uid=elasticsearch,gid=elasticsearch,endpoint=eu-central-1,dbglevel=debug 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][mask=F_OK]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][flags=0x8000]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:  [path=/]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      [tpath=/]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/] []
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      url is https://s3.amazonaws.com
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      HTTP response code 200
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:  [path=/][list=0]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      [count=0]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis][mask=F_OK]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis][flags=0x8000]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:  [path=/analysis]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      [tpath=/analysis]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/analysis/
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/analysis/
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/analysis/] []
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      url is https://s3.amazonaws.com
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      HTTP response code 200
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:  [path=/analysis/][list=0]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]:      [count=0]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/custom_stemmer_48a0c3c6a534a02b91120196cc2858e8]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/custom_stemmer_48a0c3c6a534a02b91120196cc2858e8][mask=F_OK]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_2b3fee32c0fd29436d4b6856b5edf01e]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_2b3fee32c0fd29436d4b6856b5edf01e][mask=F_OK]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_c0d7655dfcfe9983684b4b13bee3b9d1]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_c0d7655dfcfe9983684b4b13bee3b9d1][mask=F_OK]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][mask=F_OK]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][flags=0x8000]
Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:  [path=/]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      [tpath=/]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/] []
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      url is https://s3.amazonaws.com
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      HTTP response code 200
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:  [path=/][list=0]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      [count=0]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis][mask=F_OK]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis][flags=0x8000]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:  [path=/analysis]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      [tpath=/analysis]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/analysis/
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/analysis/
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/analysis/] []
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      url is https://s3.amazonaws.com
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      HTTP response code 200
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:  [path=/analysis/][list=0]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:      [count=0]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/custom_stemmer_48a0c3c6a534a02b91120196cc2858e8][mask=F_OK]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_2b3fee32c0fd29436d4b6856b5edf01e][mask=F_OK]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_c0d7655dfcfe9983684b4b13bee3b9d1][mask=F_OK]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/][mask=F_OK]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/][flags=0x8000]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/]
Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]:  [path=/]
...

Details about issue

Hi,

My bucket has something like 6 files in 2 directories, but every 5 seconds s3fs makes many calls to s3 which are super expensive. About 1k call per minute. I've edited the updatedb.conf with the following:

# not sure if the prunefs is necessary but  I've seen it on other issues.
PRUNEFS = "s3fs fuse.s3fs 9p afs anon_inodefs auto autofs bdev binfmt_misc cgroup cifs coda configfs cpuset debugfs devpts ecryptfs exofs fuse fuse.sshfs fusectl gfs gfs2 gpfs hugetlbfs inotifyfs iso9660 jffs2 lustre mqueue ncpfs nfs nfs4 nfsd pipefs proc ramfs rootfs rpc_pipefs securityfs selinuxfs sfs sockfs sysfs tmpfs ubifs udf usbfs fuse.glusterfs ceph fuse.ceph"

PRUNEPATHS = "/etc/elasticsearch/shared /afs /media /mnt /net /sfs /tmp /udev /var/cache/ccache /var/lib/yum/yumdb /var/spool/cups /var/spool/squid /var/tmp /var/lib/ceph"

But it does not reduce the number of calls.

Originally created by @JeremieDoctrine on GitHub (Mar 18, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1607 #### Version of s3fs being used (s3fs --version) 1.89 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.2 #### Kernel information (uname -r) 4.14.88-88.76.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" ``` #### /etc/fstab entry, if applicable ``` s3fs#elasticsearch-mounted-volume:/dev/ /etc/elasticsearch/shared fuse _netdev,allow_other,use_cache=/tmp/s3fs,iam_role=auto,url=https://s3.amazonaws.com,umask=0007,uid=elasticsearch,gid=elasticsearch,endpoint=eu-central-1,dbglevel=debug 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][mask=F_OK] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][flags=0x8000] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [tpath=/] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/ Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/ Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/] [] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: url is https://s3.amazonaws.com Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: HTTP response code 200 Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][list=0] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [count=0] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis][mask=F_OK] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis][flags=0x8000] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [tpath=/analysis] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/analysis/ Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/analysis/ Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/analysis/] [] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: url is https://s3.amazonaws.com Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: HTTP response code 200 Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/][list=0] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [count=0] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/custom_stemmer_48a0c3c6a534a02b91120196cc2858e8] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/custom_stemmer_48a0c3c6a534a02b91120196cc2858e8][mask=F_OK] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_2b3fee32c0fd29436d4b6856b5edf01e] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_2b3fee32c0fd29436d4b6856b5edf01e][mask=F_OK] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_c0d7655dfcfe9983684b4b13bee3b9d1] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_c0d7655dfcfe9983684b4b13bee3b9d1][mask=F_OK] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][mask=F_OK] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/][flags=0x8000] Mar 18 12:50:45 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [tpath=/] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/ Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/ Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/] [] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: url is https://s3.amazonaws.com Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: HTTP response code 200 Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/][list=0] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [count=0] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis][mask=F_OK] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis][flags=0x8000] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [tpath=/analysis] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: URL is https://s3.amazonaws.com/elasticsearch-mounted-volume?delimiter=/&max-keys=1000&prefix=dev/analysis/ Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: URL changed is https://elasticsearch-mounted-volume.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=dev/analysis/ Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=dev/analysis/] [] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: url is https://s3.amazonaws.com Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: HTTP response code 200 Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/][list=0] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [count=0] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/custom_stemmer_48a0c3c6a534a02b91120196cc2858e8][mask=F_OK] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_2b3fee32c0fd29436d4b6856b5edf01e][mask=F_OK] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/analysis/synonym_c0d7655dfcfe9983684b4b13bee3b9d1][mask=F_OK] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/][mask=F_OK] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/][flags=0x8000] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/] Mar 18 12:50:46 ip-172-31-59-43 s3fs[14934]: [path=/] ... ``` ### Details about issue Hi, My bucket has something like 6 files in 2 directories, but every 5 seconds s3fs makes many calls to s3 which are super expensive. About 1k call per minute. I've edited the `updatedb.conf` with the following: ``` # not sure if the prunefs is necessary but I've seen it on other issues. PRUNEFS = "s3fs fuse.s3fs 9p afs anon_inodefs auto autofs bdev binfmt_misc cgroup cifs coda configfs cpuset debugfs devpts ecryptfs exofs fuse fuse.sshfs fusectl gfs gfs2 gpfs hugetlbfs inotifyfs iso9660 jffs2 lustre mqueue ncpfs nfs nfs4 nfsd pipefs proc ramfs rootfs rpc_pipefs securityfs selinuxfs sfs sockfs sysfs tmpfs ubifs udf usbfs fuse.glusterfs ceph fuse.ceph" PRUNEPATHS = "/etc/elasticsearch/shared /afs /media /mnt /net /sfs /tmp /udev /var/cache/ccache /var/lib/yum/yumdb /var/spool/cups /var/spool/squid /var/tmp /var/lib/ceph" ``` But it does not reduce the number of calls.
kerem closed this issue 2026-03-04 01:49:18 +03:00
Author
Owner

@JeremieDoctrine commented on GitHub (Mar 18, 2021):

Looks like there is a process in Elasticsearch scanning /etc/elasticsearch/shared every five seconds. If I stop Elasticsearch I don't have any call anymore.

Is there any setting to tell s3fs to stop calling s3 when doing a scan on a directory?

<!-- gh-comment-id:802060829 --> @JeremieDoctrine commented on GitHub (Mar 18, 2021): Looks like there is a process in Elasticsearch scanning `/etc/elasticsearch/shared` every five seconds. If I stop Elasticsearch I don't have any call anymore. Is there any setting to tell s3fs to stop calling s3 when doing a scan on a directory?
Author
Owner

@gaul commented on GitHub (Mar 21, 2021):

I am not sure what you expect here -- a filesystem serves data when a process requests it. It is up to the application not to request that data. You can either configure Elasticsearch correctly or use some kind of unix permissions to prohibit access.

<!-- gh-comment-id:803582267 --> @gaul commented on GitHub (Mar 21, 2021): I am not sure what you expect here -- a filesystem serves data when a process requests it. It is up to the application not to request that data. You can either configure Elasticsearch correctly or use some kind of unix permissions to prohibit access.
Author
Owner

@gaul commented on GitHub (Apr 25, 2021):

Closing since there is nothing actionable on the s3fs side.

<!-- gh-comment-id:826247699 --> @gaul commented on GitHub (Apr 25, 2021): Closing since there is nothing actionable on the s3fs side.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#846
No description provided.