starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2125] IO error after write about 100,000 files #1081

New issue
Closed
opened 2026-03-04 01:51:14 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:51:14 +03:00
Owner
Copy link

Originally created by @huntersman on GitHub (Mar 10, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2125

Additional Information

Version of s3fs being used (s3fs --version)

V1.91(commit:unknown)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.2

Kernel information (uname -r)

5.4.213-1.el7.elrepo.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

CentOS7

How to run s3fs, if applicable

s3fs demo /root/demo -o passwd_file=${HOME}/.passwd-s3fs -o url=http://xxxx -o use_path_request_style -o noxmlns -o dbglevel=debug -o default_acl=public-read -o logfile=/var/log/s3fs.log -o allow_other -o multireq_max=500 -o nocopyapi -o use_cache="/root/s3fs/" -o del_cache -o parallel_count=500

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

As debug messages are very large, I only put last part of it.

2023-03-10T06:26:26.762Z [INF] s3fs.cpp:s3fs_getattr(893): [path=/myfile.txt]
2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:check_parent_object_access(743): [path=/myfile.txt]
2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:check_object_access(635): [path=/]
2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:check_object_access(640): [pid=2375,uid=0,gid=0]
2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/]
2023-03-10T06:26:26.763Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/][time=19847.739981957][hit count=815]
2023-03-10T06:26:26.763Z [DBG] s3fs.cpp:check_object_access(635): [path=/myfile.txt]
2023-03-10T06:26:26.763Z [DBG] s3fs.cpp:check_object_access(640): [pid=2375,uid=0,gid=0]
2023-03-10T06:26:26.763Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/myfile.txt]
2023-03-10T06:26:26.763Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/myfile.txt][time=19045.401991495][hit count=11819]
2023-03-10T06:26:26.763Z [DBG] fdcache.cpp:OpenExistFdEntity(645): [path=/myfile.txt][flags=0x0]
2023-03-10T06:26:26.764Z [DBG] fdcache.cpp:Open(537): [path=/myfile.txt][size=-1][ts_mctime=0.1073741822][flags=0x0][force_tmpfile=no][create=no][ignore_modify=no]
2023-03-10T06:26:26.764Z [DBG] fdcache_entity.cpp:Open(428): [path=/myfile.txt][physical_fd=10][size=-1][ts_mctime=0.1073741822][flags=0x0]
2023-03-10T06:26:26.764Z [DBG] s3fs.cpp:s3fs_getattr(916): [path=/myfile.txt] uid=0, gid=0, mode=100744
2023-03-10T06:26:26.764Z [DBG] fdcache.cpp:Close(720): [ent->file=/myfile.txt][pseudo_fd=4]
2023-03-10T06:26:26.764Z [DBG] fdcache_entity.cpp:Close(218): [path=/myfile.txt][pseudo_fd=4][physical_fd=10]
2023-03-10T06:26:29.130Z [INF] s3fs.cpp:s3fs_flush(2853): [path=/cp.log][pseudo_fd=3]
2023-03-10T06:26:29.130Z [DBG] s3fs.cpp:check_parent_object_access(743): [path=/cp.log]
2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:check_object_access(635): [path=/]
2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0]
2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/]
2023-03-10T06:26:29.131Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/][time=19847.739981957][hit count=816]
2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:check_object_access(635): [path=/cp.log]
2023-03-10T06:26:29.132Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0]
2023-03-10T06:26:29.132Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/cp.log]
2023-03-10T06:26:29.132Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/cp.log][time=19044.839991501][hit count=5914]
2023-03-10T06:26:29.132Z [DBG] fdcache.cpp:GetExistFdEntity(628): [path=/cp.log][pseudo_fd=3]
2023-03-10T06:26:29.132Z [INF]       fdcache_entity.cpp:RowFlush(1404): [tpath=][path=/cp.log][pseudo_fd=3][physical_fd=11]
2023-03-10T06:26:29.132Z [DBG] fdcache_page.cpp:Dump(984): pages (shrinked=no) = {
2023-03-10T06:26:29.133Z [DBG] fdcache_page.cpp:Dump(986):   [00000000] -> {00000000000000 - 00000011414262 : loaded / modified}
2023-03-10T06:26:29.133Z [DBG] fdcache_page.cpp:Dump(988): }
2023-03-10T06:26:29.133Z [INF]       fdcache_entity.cpp:RowFlushMultipart(1522): [tpath=][path=/cp.log][pseudo_fd=3][physical_fd=11]
2023-03-10T06:26:29.133Z [DBG] fdcache_entity.cpp:Load(1071): [path=/cp.log][physical_fd=11][offset=0][size=0]
2023-03-10T06:26:29.134Z [INF]       curl.cpp:PutRequest(3297): [tpath=/cp.log]
2023-03-10T06:26:29.134Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-03-10T06:26:29.134Z [INF]       curl_util.cpp:prepare_url(257): URL is http://xxxx/demo/cp.log
2023-03-10T06:26:29.134Z [INF]       curl_util.cpp:prepare_url(290): URL changed is http://xxxx/demo/cp.log
2023-03-10T06:26:29.134Z [INF]       curl.cpp:PutRequest(3425): uploading... [path=/cp.log][fd=11][size=11414262]
2023-03-10T06:26:29.134Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://xxxx/demo/cp.log
2023-03-10T06:26:29.173Z [INF]       curl.cpp:insertV4Headers(2737): computing signature [PUT] [/cp.log] [] [0b683b2abcd280de95be19df7d8bdc1de3485f2a840b766e859d99237c9e171c]
2023-03-10T06:26:29.173Z [INF]       curl_util.cpp:url_to_host(334): url is http://xxxx
2023-03-10T06:26:29.565Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-03-10T06:26:29.565Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-03-10T06:26:29.566Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/cp.log]
2023-03-10T06:26:29.566Z [INF] s3fs.cpp:s3fs_flush(2853): [path=/myfile.txt][pseudo_fd=2]
2023-03-10T06:26:29.566Z [DBG] s3fs.cpp:check_parent_object_access(743): [path=/myfile.txt]
2023-03-10T06:26:29.566Z [DBG] s3fs.cpp:check_object_access(635): [path=/]
2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0]
2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/]
2023-03-10T06:26:29.567Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/][time=19847.739981957][hit count=817]
2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:check_object_access(635): [path=/myfile.txt]
2023-03-10T06:26:29.567Z [INF] s3fs.cpp:s3fs_release(2931): [path=/cp.log][pseudo_fd=3]
2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0]
2023-03-10T06:26:29.568Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/myfile.txt]
2023-03-10T06:26:29.568Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/cp.log]
2023-03-10T06:26:29.568Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/myfile.txt][time=19045.401991495][hit count=11820]
2023-03-10T06:26:29.568Z [DBG] fdcache.cpp:GetExistFdEntity(628): [path=/myfile.txt][pseudo_fd=2]
2023-03-10T06:26:29.568Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/cp.log][pseudo_fd=3]
2023-03-10T06:26:29.568Z [INF]       fdcache_entity.cpp:RowFlush(1404): [tpath=][path=/myfile.txt][pseudo_fd=2][physical_fd=10]
2023-03-10T06:26:29.569Z [DBG] fdcache_page.cpp:Dump(984): pages (shrinked=no) = {
2023-03-10T06:26:29.569Z [DBG] fdcache_page.cpp:Dump(986):   [00000000] -> {00000000000000 - 00000005292766 : loaded / modified}
2023-03-10T06:26:29.569Z [DBG] fdcache_page.cpp:Dump(988): }
2023-03-10T06:26:29.569Z [INF]       fdcache_entity.cpp:RowFlushMultipart(1522): [tpath=][path=/myfile.txt][pseudo_fd=2][physical_fd=10]
2023-03-10T06:26:29.569Z [DBG] fdcache.cpp:Close(720): [ent->file=/cp.log][pseudo_fd=3]
2023-03-10T06:26:29.570Z [DBG] fdcache_entity.cpp:Load(1071): [path=/myfile.txt][physical_fd=10][offset=0][size=0]
2023-03-10T06:26:29.570Z [INF]       curl.cpp:PutRequest(3297): [tpath=/myfile.txt]
2023-03-10T06:26:29.570Z [DBG] fdcache_entity.cpp:Close(218): [path=/cp.log][pseudo_fd=3][physical_fd=11]
2023-03-10T06:26:29.570Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-03-10T06:26:29.570Z [DBG] fdcache_stat.cpp:RawOpen(235): file locked(/cp.log - /root/s3fs//.demo.stat/cp.log)
2023-03-10T06:26:29.570Z [INF]       curl_util.cpp:prepare_url(257): URL is http://xxxx/demo/myfile.txt
2023-03-10T06:26:29.571Z [INF]       curl_util.cpp:prepare_url(290): URL changed is http://xxxx/demo/myfile.txt
2023-03-10T06:26:29.571Z [DBG] fdcache_stat.cpp:Release(261): file unlocked(/cp.log)
2023-03-10T06:26:29.571Z [INF]       curl.cpp:PutRequest(3425): uploading... [path=/myfile.txt][fd=10][size=5292766]
2023-03-10T06:26:29.571Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://xxxx/demo/myfile.txt
2023-03-10T06:26:29.571Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/cp.log][pseudo_fd=-1]
2023-03-10T06:26:29.594Z [INF]       curl.cpp:insertV4Headers(2737): computing signature [PUT] [/myfile.txt] [] [010ffe615194b4a4d02eb47754fb3659796d5a36bdd40f1f92a7274ce2d0ebc1]
2023-03-10T06:26:29.594Z [INF]       curl_util.cpp:url_to_host(334): url is http://xxxx
2023-03-10T06:26:29.798Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-03-10T06:26:29.798Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-03-10T06:26:29.798Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/myfile.txt]
2023-03-10T06:26:29.800Z [INF] s3fs.cpp:s3fs_release(2931): [path=/myfile.txt][pseudo_fd=2]
2023-03-10T06:26:29.800Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/myfile.txt]
2023-03-10T06:26:29.801Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/myfile.txt][pseudo_fd=2]
2023-03-10T06:26:29.801Z [DBG] fdcache.cpp:Close(720): [ent->file=/myfile.txt][pseudo_fd=2]
2023-03-10T06:26:29.801Z [DBG] fdcache_entity.cpp:Close(218): [path=/myfile.txt][pseudo_fd=2][physical_fd=10]
2023-03-10T06:26:29.801Z [DBG] fdcache_stat.cpp:RawOpen(235): file locked(/myfile.txt - /root/s3fs//.demo.stat/myfile.txt)
2023-03-10T06:26:29.801Z [DBG] fdcache_stat.cpp:Release(261): file unlocked(/myfile.txt)
2023-03-10T06:26:29.802Z [INF]       fdcache.cpp:GetFdEntity(484): [path=/myfile.txt][pseudo_fd=-1]

Details about issue

I upload a lot of small size files to s3fs and find out s3fs will eventually become unusable.

ll /root/demo
ls: cannot access /root/demo: Input/output error

Usually it happens after 100,000 files are written. I wonder know why s3fs appears IO error when write a lot of files. Thank you for your help!

Originally created by @huntersman on GitHub (Mar 10, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2125 <!-- -------------------------------------------------------------------------- The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD. --------------------------------------------------------------------------- --> ### Additional Information #### Version of s3fs being used (`s3fs --version`) <!-- example: V1.91 (commit:b19262a) --> V1.91(commit:unknown) #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) <!-- example: 2.9.2 --> 2.9.2 #### Kernel information (`uname -r`) <!-- example: 5.10.96-90.460.amzn2.x86_64 --> 5.4.213-1.el7.elrepo.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) <!-- command result --> CentOS7 #### How to run s3fs, if applicable <!-- Describe the s3fs "command line" or "/etc/fstab" entry used. --> <!-- Executed command line or /etc/fastab entry --> ``` s3fs demo /root/demo -o passwd_file=${HOME}/.passwd-s3fs -o url=http://xxxx -o use_path_request_style -o noxmlns -o dbglevel=debug -o default_acl=public-read -o logfile=/var/log/s3fs.log -o allow_other -o multireq_max=500 -o nocopyapi -o use_cache="/root/s3fs/" -o del_cache -o parallel_count=500 ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) <!-- if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages. --> As debug messages are very large, I only put last part of it. ``` 2023-03-10T06:26:26.762Z [INF] s3fs.cpp:s3fs_getattr(893): [path=/myfile.txt] 2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:check_parent_object_access(743): [path=/myfile.txt] 2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:check_object_access(635): [path=/] 2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:check_object_access(640): [pid=2375,uid=0,gid=0] 2023-03-10T06:26:26.762Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/] 2023-03-10T06:26:26.763Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/][time=19847.739981957][hit count=815] 2023-03-10T06:26:26.763Z [DBG] s3fs.cpp:check_object_access(635): [path=/myfile.txt] 2023-03-10T06:26:26.763Z [DBG] s3fs.cpp:check_object_access(640): [pid=2375,uid=0,gid=0] 2023-03-10T06:26:26.763Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/myfile.txt] 2023-03-10T06:26:26.763Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/myfile.txt][time=19045.401991495][hit count=11819] 2023-03-10T06:26:26.763Z [DBG] fdcache.cpp:OpenExistFdEntity(645): [path=/myfile.txt][flags=0x0] 2023-03-10T06:26:26.764Z [DBG] fdcache.cpp:Open(537): [path=/myfile.txt][size=-1][ts_mctime=0.1073741822][flags=0x0][force_tmpfile=no][create=no][ignore_modify=no] 2023-03-10T06:26:26.764Z [DBG] fdcache_entity.cpp:Open(428): [path=/myfile.txt][physical_fd=10][size=-1][ts_mctime=0.1073741822][flags=0x0] 2023-03-10T06:26:26.764Z [DBG] s3fs.cpp:s3fs_getattr(916): [path=/myfile.txt] uid=0, gid=0, mode=100744 2023-03-10T06:26:26.764Z [DBG] fdcache.cpp:Close(720): [ent->file=/myfile.txt][pseudo_fd=4] 2023-03-10T06:26:26.764Z [DBG] fdcache_entity.cpp:Close(218): [path=/myfile.txt][pseudo_fd=4][physical_fd=10] 2023-03-10T06:26:29.130Z [INF] s3fs.cpp:s3fs_flush(2853): [path=/cp.log][pseudo_fd=3] 2023-03-10T06:26:29.130Z [DBG] s3fs.cpp:check_parent_object_access(743): [path=/cp.log] 2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:check_object_access(635): [path=/] 2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0] 2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/] 2023-03-10T06:26:29.131Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/][time=19847.739981957][hit count=816] 2023-03-10T06:26:29.131Z [DBG] s3fs.cpp:check_object_access(635): [path=/cp.log] 2023-03-10T06:26:29.132Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0] 2023-03-10T06:26:29.132Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/cp.log] 2023-03-10T06:26:29.132Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/cp.log][time=19044.839991501][hit count=5914] 2023-03-10T06:26:29.132Z [DBG] fdcache.cpp:GetExistFdEntity(628): [path=/cp.log][pseudo_fd=3] 2023-03-10T06:26:29.132Z [INF] fdcache_entity.cpp:RowFlush(1404): [tpath=][path=/cp.log][pseudo_fd=3][physical_fd=11] 2023-03-10T06:26:29.132Z [DBG] fdcache_page.cpp:Dump(984): pages (shrinked=no) = { 2023-03-10T06:26:29.133Z [DBG] fdcache_page.cpp:Dump(986): [00000000] -> {00000000000000 - 00000011414262 : loaded / modified} 2023-03-10T06:26:29.133Z [DBG] fdcache_page.cpp:Dump(988): } 2023-03-10T06:26:29.133Z [INF] fdcache_entity.cpp:RowFlushMultipart(1522): [tpath=][path=/cp.log][pseudo_fd=3][physical_fd=11] 2023-03-10T06:26:29.133Z [DBG] fdcache_entity.cpp:Load(1071): [path=/cp.log][physical_fd=11][offset=0][size=0] 2023-03-10T06:26:29.134Z [INF] curl.cpp:PutRequest(3297): [tpath=/cp.log] 2023-03-10T06:26:29.134Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-03-10T06:26:29.134Z [INF] curl_util.cpp:prepare_url(257): URL is http://xxxx/demo/cp.log 2023-03-10T06:26:29.134Z [INF] curl_util.cpp:prepare_url(290): URL changed is http://xxxx/demo/cp.log 2023-03-10T06:26:29.134Z [INF] curl.cpp:PutRequest(3425): uploading... [path=/cp.log][fd=11][size=11414262] 2023-03-10T06:26:29.134Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://xxxx/demo/cp.log 2023-03-10T06:26:29.173Z [INF] curl.cpp:insertV4Headers(2737): computing signature [PUT] [/cp.log] [] [0b683b2abcd280de95be19df7d8bdc1de3485f2a840b766e859d99237c9e171c] 2023-03-10T06:26:29.173Z [INF] curl_util.cpp:url_to_host(334): url is http://xxxx 2023-03-10T06:26:29.565Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-03-10T06:26:29.565Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-03-10T06:26:29.566Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/cp.log] 2023-03-10T06:26:29.566Z [INF] s3fs.cpp:s3fs_flush(2853): [path=/myfile.txt][pseudo_fd=2] 2023-03-10T06:26:29.566Z [DBG] s3fs.cpp:check_parent_object_access(743): [path=/myfile.txt] 2023-03-10T06:26:29.566Z [DBG] s3fs.cpp:check_object_access(635): [path=/] 2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0] 2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/] 2023-03-10T06:26:29.567Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/][time=19847.739981957][hit count=817] 2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:check_object_access(635): [path=/myfile.txt] 2023-03-10T06:26:29.567Z [INF] s3fs.cpp:s3fs_release(2931): [path=/cp.log][pseudo_fd=3] 2023-03-10T06:26:29.567Z [DBG] s3fs.cpp:check_object_access(640): [pid=39339,uid=0,gid=0] 2023-03-10T06:26:29.568Z [DBG] s3fs.cpp:get_object_attribute(403): [path=/myfile.txt] 2023-03-10T06:26:29.568Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/cp.log] 2023-03-10T06:26:29.568Z [DBG] cache.cpp:GetStat(265): stat cache hit [path=/myfile.txt][time=19045.401991495][hit count=11820] 2023-03-10T06:26:29.568Z [DBG] fdcache.cpp:GetExistFdEntity(628): [path=/myfile.txt][pseudo_fd=2] 2023-03-10T06:26:29.568Z [INF] fdcache.cpp:GetFdEntity(484): [path=/cp.log][pseudo_fd=3] 2023-03-10T06:26:29.568Z [INF] fdcache_entity.cpp:RowFlush(1404): [tpath=][path=/myfile.txt][pseudo_fd=2][physical_fd=10] 2023-03-10T06:26:29.569Z [DBG] fdcache_page.cpp:Dump(984): pages (shrinked=no) = { 2023-03-10T06:26:29.569Z [DBG] fdcache_page.cpp:Dump(986): [00000000] -> {00000000000000 - 00000005292766 : loaded / modified} 2023-03-10T06:26:29.569Z [DBG] fdcache_page.cpp:Dump(988): } 2023-03-10T06:26:29.569Z [INF] fdcache_entity.cpp:RowFlushMultipart(1522): [tpath=][path=/myfile.txt][pseudo_fd=2][physical_fd=10] 2023-03-10T06:26:29.569Z [DBG] fdcache.cpp:Close(720): [ent->file=/cp.log][pseudo_fd=3] 2023-03-10T06:26:29.570Z [DBG] fdcache_entity.cpp:Load(1071): [path=/myfile.txt][physical_fd=10][offset=0][size=0] 2023-03-10T06:26:29.570Z [INF] curl.cpp:PutRequest(3297): [tpath=/myfile.txt] 2023-03-10T06:26:29.570Z [DBG] fdcache_entity.cpp:Close(218): [path=/cp.log][pseudo_fd=3][physical_fd=11] 2023-03-10T06:26:29.570Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-03-10T06:26:29.570Z [DBG] fdcache_stat.cpp:RawOpen(235): file locked(/cp.log - /root/s3fs//.demo.stat/cp.log) 2023-03-10T06:26:29.570Z [INF] curl_util.cpp:prepare_url(257): URL is http://xxxx/demo/myfile.txt 2023-03-10T06:26:29.571Z [INF] curl_util.cpp:prepare_url(290): URL changed is http://xxxx/demo/myfile.txt 2023-03-10T06:26:29.571Z [DBG] fdcache_stat.cpp:Release(261): file unlocked(/cp.log) 2023-03-10T06:26:29.571Z [INF] curl.cpp:PutRequest(3425): uploading... [path=/myfile.txt][fd=10][size=5292766] 2023-03-10T06:26:29.571Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://xxxx/demo/myfile.txt 2023-03-10T06:26:29.571Z [INF] fdcache.cpp:GetFdEntity(484): [path=/cp.log][pseudo_fd=-1] 2023-03-10T06:26:29.594Z [INF] curl.cpp:insertV4Headers(2737): computing signature [PUT] [/myfile.txt] [] [010ffe615194b4a4d02eb47754fb3659796d5a36bdd40f1f92a7274ce2d0ebc1] 2023-03-10T06:26:29.594Z [INF] curl_util.cpp:url_to_host(334): url is http://xxxx 2023-03-10T06:26:29.798Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-03-10T06:26:29.798Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-03-10T06:26:29.798Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/myfile.txt] 2023-03-10T06:26:29.800Z [INF] s3fs.cpp:s3fs_release(2931): [path=/myfile.txt][pseudo_fd=2] 2023-03-10T06:26:29.800Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/myfile.txt] 2023-03-10T06:26:29.801Z [INF] fdcache.cpp:GetFdEntity(484): [path=/myfile.txt][pseudo_fd=2] 2023-03-10T06:26:29.801Z [DBG] fdcache.cpp:Close(720): [ent->file=/myfile.txt][pseudo_fd=2] 2023-03-10T06:26:29.801Z [DBG] fdcache_entity.cpp:Close(218): [path=/myfile.txt][pseudo_fd=2][physical_fd=10] 2023-03-10T06:26:29.801Z [DBG] fdcache_stat.cpp:RawOpen(235): file locked(/myfile.txt - /root/s3fs//.demo.stat/myfile.txt) 2023-03-10T06:26:29.801Z [DBG] fdcache_stat.cpp:Release(261): file unlocked(/myfile.txt) 2023-03-10T06:26:29.802Z [INF] fdcache.cpp:GetFdEntity(484): [path=/myfile.txt][pseudo_fd=-1] ``` ### Details about issue <!-- Please describe the content of the issue in detail. --> I upload a lot of small size files to s3fs and find out s3fs will eventually become unusable. ``` ll /root/demo ls: cannot access /root/demo: Input/output error ``` Usually it happens after 100,000 files are written. I wonder know why s3fs appears IO error when write a lot of files. Thank you for your help!
kerem closed this issue 2026-03-04 01:51:15 +03:00
kerem commented 2026-03-04 01:51:15 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 12, 2023):

Does s3fs not terminate in this case and does s3fs is doing any communication or processing?

You may need to adjust the max_stat_cache_size, stat_cache_expire, stat_cache_interval_expire etc options.
(Are 100,000 files in the same /root/demo directory?)

Also I'm wonder about the use_cache option.
I assume there are 100,000 files(as cache files) in the cache directory.
It's hard to believe that this cache files affect the ls command, but I can imagine that having 100,000 files in one local directory would have a big impact.(but I'm not sure that these many files effect to this issue.)

<!-- gh-comment-id:1465170769 --> @ggtakec commented on GitHub (Mar 12, 2023): Does s3fs not terminate in this case and does s3fs is doing any communication or processing? You may need to adjust `the max_stat_cache_size`, `stat_cache_expire`, `stat_cache_interval_expire` etc options. _(Are 100,000 files in the same `/root/demo` directory?)_ Also I'm wonder about the `use_cache` option. I assume there are 100,000 files(as cache files) in the cache directory. It's hard to believe that this cache files affect the `ls` command, but I can imagine that having 100,000 files in one local directory would have a big impact.(but I'm not sure that these many files effect to this issue.)
kerem commented 2026-03-04 01:51:15 +03:00
Author
Owner
Copy link

@huntersman commented on GitHub (Mar 13, 2023):

The s3fs process does not exit but it seems to be stuck, and I can't get any response from it.

Are 100,000 files in the same /root/demo directory?

Yes, they are in the same directory.

<!-- gh-comment-id:1465358015 --> @huntersman commented on GitHub (Mar 13, 2023): The s3fs process does not exit but it seems to be stuck, and I can't get any response from it. > Are 100,000 files in the same /root/demo directory? Yes, they are in the same directory.
kerem commented 2026-03-04 01:51:15 +03:00
Author
Owner
Copy link

@huntersman commented on GitHub (Mar 14, 2023):

@ggtakec Thank you very much. I set max_stat_cache_size=1000000 and stat_cache_expire=300, IO error never happens again.

<!-- gh-comment-id:1467461762 --> @huntersman commented on GitHub (Mar 14, 2023): @ggtakec Thank you very much. I set `max_stat_cache_size=1000000` and `stat_cache_expire=300`, IO error never happens again.
kerem commented 2026-03-04 01:51:16 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 16, 2023):

@huntersman Thanks for confirming.
I think that the stat cache overflowed and the error occurred because the stat information was reacquired every time.

<!-- gh-comment-id:1472076329 --> @ggtakec commented on GitHub (Mar 16, 2023): @huntersman Thanks for confirming. I think that the stat cache overflowed and the error occurred because the stat information was reacquired every time.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1081
No description provided.