[GH-ISSUE #2446] Too many del stat cache entry and head requests for just 1 file requested #1205

New issue

Open

opened 2026-03-04 01:52:12 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 01:52:12 +03:00

Owner

Originally created by @nguyenminhdungpg on GitHub (Apr 19, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2446

Additional Information

Version of s3fs being used (`s3fs --version`)

Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

2.9.9

Kernel information (`uname -r`)

6.2.16-19-pve

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"

How to run s3fs, if applicable

[] command line
s3fs $aws_bucket /home/mounted -o endpoint=ap-southeast-1 -o allow_other -o umask=000 -o dbglevel=info -o use_cache=/home/mounted-cache -o disable_noobj_cache -o max_background=1000 -o parallel_count=30 -o multireq_max=50 -o logfile=/var/log/mounted.log
[] /etc/fstab

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

2024-04-19T08:37:36.180Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5]
2024-04-19T08:37:36.179Z [INF] s3fs.cpp:s3fs_release(3062): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5]
2024-04-19T08:37:36.179Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.179Z [INF]       fdcache_entity.cpp:RowFlush(1413): [tpath=][path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5][physical_fd=26]
2024-04-19T08:37:36.179Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.108Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:36.108Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:36.108Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:36.108Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:36.108Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.107Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5]
2024-04-19T08:37:36.060Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4]
2024-04-19T08:37:36.060Z [INF] s3fs.cpp:s3fs_release(3062): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4]
2024-04-19T08:37:36.057Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.057Z [INF]       fdcache_entity.cpp:RowFlush(1413): [tpath=][path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4][physical_fd=26]
2024-04-19T08:37:36.057Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4]
2024-04-19T08:37:36.019Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.014Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.014Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:36.010Z [INF]       fdcache_entity.cpp:SetMCtime(835): [path=/Upload/img_2024_04_11_10_39_47.jpg][physical_fd=26][mtime=1712807063][ctime=1712807063]
2024-04-19T08:37:36.008Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.936Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:35.936Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.936Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.936Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:35.936Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.935Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.935Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=-1]
2024-04-19T08:37:35.935Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:35.935Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.935Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.935Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:35.935Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.934Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.934Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=-1]
2024-04-19T08:37:35.931Z [INF] s3fs.cpp:s3fs_open(2805): [path=/Upload/img_2024_04_11_10_39_47.jpg][flags=0x8800]
2024-04-19T08:37:35.931Z [INF] s3fs.cpp:s3fs_open(2805): [path=/Upload/img_2024_04_11_10_39_47.jpg][flags=0x8800]
2024-04-19T08:37:35.930Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.854Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] []
2024-04-19T08:37:35.854Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.853Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg
2024-04-19T08:37:35.852Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084]
2024-04-19T08:37:35.852Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg]
2024-04-19T08:37:35.852Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/Upload/img_2024_04_11_10_39_47.jpg]

Details about issue

In haproxy log, there is only one request to img_2024_04_11_10_39_47.jpg but in logs of s3fs, there are 4 times of Head Request sent to S3 and 4 times executed delete stat cache entry of the file.
My website has very large number of uploaded images and when it takes high traffics, s3fs is very slow and there so so much of Head requests like above.
Could you please tell me why there are so much Head requests and how to optimize this.
Thank you very much.

Originally created by @nguyenminhdungpg on GitHub (Apr 19, 2024). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2446  ### Additional Information #### Version of s3fs being used (`s3fs --version`)  Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)  2.9.9 #### Kernel information (`uname -r`)  6.2.16-19-pve #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`)  NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" #### How to run s3fs, if applicable  [] command line s3fs $aws_bucket /home/mounted -o endpoint=ap-southeast-1 -o allow_other -o umask=000 -o dbglevel=info -o use_cache=/home/mounted-cache -o disable_noobj_cache -o max_background=1000 -o parallel_count=30 -o multireq_max=50 -o logfile=/var/log/mounted.log [] /etc/fstab  ``` ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` 2024-04-19T08:37:36.180Z [INF] fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5] 2024-04-19T08:37:36.179Z [INF] s3fs.cpp:s3fs_release(3062): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5] 2024-04-19T08:37:36.179Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.179Z [INF] fdcache_entity.cpp:RowFlush(1413): [tpath=][path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5][physical_fd=26] 2024-04-19T08:37:36.179Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.108Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] [] 2024-04-19T08:37:36.108Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:36.108Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:36.108Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084] 2024-04-19T08:37:36.108Z [INF] curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.107Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=5] 2024-04-19T08:37:36.060Z [INF] fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4] 2024-04-19T08:37:36.060Z [INF] s3fs.cpp:s3fs_release(3062): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4] 2024-04-19T08:37:36.057Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.057Z [INF] fdcache_entity.cpp:RowFlush(1413): [tpath=][path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4][physical_fd=26] 2024-04-19T08:37:36.057Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=4] 2024-04-19T08:37:36.019Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.014Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.014Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:36.010Z [INF] fdcache_entity.cpp:SetMCtime(835): [path=/Upload/img_2024_04_11_10_39_47.jpg][physical_fd=26][mtime=1712807063][ctime=1712807063] 2024-04-19T08:37:36.008Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.936Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] [] 2024-04-19T08:37:35.936Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:35.936Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:35.936Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084] 2024-04-19T08:37:35.936Z [INF] curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.935Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.935Z [INF] fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=-1] 2024-04-19T08:37:35.935Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] [] 2024-04-19T08:37:35.935Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:35.935Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:35.935Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084] 2024-04-19T08:37:35.935Z [INF] curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.934Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.934Z [INF] fdcache.cpp:GetFdEntity(484): [path=/Upload/img_2024_04_11_10_39_47.jpg][pseudo_fd=-1] 2024-04-19T08:37:35.931Z [INF] s3fs.cpp:s3fs_open(2805): [path=/Upload/img_2024_04_11_10_39_47.jpg][flags=0x8800] 2024-04-19T08:37:35.931Z [INF] s3fs.cpp:s3fs_open(2805): [path=/Upload/img_2024_04_11_10_39_47.jpg][flags=0x8800] 2024-04-19T08:37:35.930Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.854Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/Upload/img_2024_04_11_10_39_47.jpg] [] [] 2024-04-19T08:37:35.854Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://image_bucket.s3.amazonaws.com/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:35.853Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/image_bucket/Upload/img_2024_04_11_10_39_47.jpg 2024-04-19T08:37:35.852Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/Upload/img_2024_04_11_10_39_47.jpg][bpath=][save=][sseckeypos=123456850850435084] 2024-04-19T08:37:35.852Z [INF] curl.cpp:HeadRequest(3299): [tpath=/Upload/img_2024_04_11_10_39_47.jpg] 2024-04-19T08:37:35.852Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/Upload/img_2024_04_11_10_39_47.jpg] ``` ### Details about issue  In haproxy log, there is only one request to img_2024_04_11_10_39_47.jpg but in logs of s3fs, there are 4 times of Head Request sent to S3 and 4 times executed delete stat cache entry of the file. My website has very large number of uploaded images and when it takes high traffics, s3fs is very slow and there so so much of Head requests like above. Could you please tell me why there are so much Head requests and how to optimize this. Thank you very much.