starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #115] Connection keeps dropping with multi-level directories #70

New issue
Closed
opened 2026-03-04 01:41:46 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:41:46 +03:00
Owner
Copy link

Originally created by @movence on GitHub (Jan 28, 2015).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/115

mounting a bucket using IAM written to /etc/passwd-s3fs

s3fs -d -f -o allow_other,use_cache=/tmp,retries=5,rw,multireq_max=7 test_bucket /mnt/s3
tried both with and without '-d -f' and got the similar problems

The leaf files are in usually 3 or 4 deep down in directory hierarchy.

sub1
  |-sub2
     |-sub3
        |-sub4
          |-files...

reading files under /mnt/s3 is fine, but the connection gets lost even doing ls -l on sub3 level. It looks like the behavior is more random as going into sub directories.

$ sudo s3fs -d -f -o allow_other,use_cache=/tmp,retries=5,rw,multireq_max=7 test_bucket /mnt/s3

    set_moutpoint_attribute(3291): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777)
s3fs_init(2595): init
s3fs_check_service(2894): check services.
    CheckBucket(2228): check a bucket.
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com/
    RequestPerform(1483): HTTP response code 200
s3fs_getattr(691): [path=/]
s3fs_getattr(691): [path=/]
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_opendir(2050): [path=/][flags=100352]
s3fs_readdir(2182): [path=/]
  list_bucket(2225): [path=/]
    ListBucketRequest(2270): [tpath=/]
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=1000
    RequestPerform(1483): HTTP response code 200
  readdir_multi_head(2105): [path=/][list=0]
    Request(3150): [count=3]
MultiRead(3113): failed to read(remaining: 2 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 2 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
    AddStat(247): add stat cache entry[path=/sub1/]
    AddStat(247): add stat cache entry[path=/test1.txt]
    AddStat(247): add stat cache entry[path=/test.txt]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478014][hit count=0]
    GetStat(170): stat cache hit [path=/test.txt][time=1422478014][hit count=0]
    GetStat(170): stat cache hit [path=/test1.txt][time=1422478014][hit count=0]
s3fs_getattr(691): [path=/sub1]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478014][hit count=1]
s3fs_getattr(691): [path=/test.txt]
    GetStat(170): stat cache hit [path=/test.txt][time=1422478014][hit count=1]
s3fs_getattr(691): [path=/test1.txt]
    GetStat(170): stat cache hit [path=/test1.txt][time=1422478014][hit count=1]
s3fs_opendir(2050): [path=/][flags=100352]
s3fs_readdir(2182): [path=/]
  list_bucket(2225): [path=/]
    ListBucketRequest(2270): [tpath=/]
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=1000
    RequestPerform(1483): HTTP response code 200
  readdir_multi_head(2105): [path=/][list=0]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478014][hit count=2]
    GetStat(170): stat cache hit [path=/test.txt][time=1422478014][hit count=2]
    GetStat(170): stat cache hit [path=/test1.txt][time=1422478014][hit count=2]
    Request(3150): [count=0]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478025][hit count=3]
    GetStat(170): stat cache hit [path=/test.txt][time=1422478025][hit count=3]
    GetStat(170): stat cache hit [path=/test1.txt][time=1422478025][hit count=3]
s3fs_getattr(691): [path=/]
s3fs_getattr(691): [path=/test1.txt]
    GetStat(170): stat cache hit [path=/test1.txt][time=1422478025][hit count=4]
s3fs_access(2646): [path=/test1.txt][mask=W_OK ]
    GetStat(170): stat cache hit [path=/test1.txt][time=1422478025][hit count=5]
s3fs_unlink(888): [path=/test1.txt]
    DeleteRequest(1787): [tpath=/test1.txt]
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com/test1.txt
    RequestPerform(1483): HTTP response code 204
    DelStat(375): delete stat cache entry[path=/test1.txt]
s3fs_getattr(691): [path=/test.txt]
    GetStat(170): stat cache hit [path=/test.txt][time=1422478025][hit count=4]
s3fs_access(2646): [path=/test.txt][mask=W_OK ]
    GetStat(170): stat cache hit [path=/test.txt][time=1422478025][hit count=5]
s3fs_unlink(888): [path=/test.txt]
    DeleteRequest(1787): [tpath=/test.txt]
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com/test.txt
    RequestPerform(1483): HTTP response code 204
    DelStat(375): delete stat cache entry[path=/test.txt]
s3fs_opendir(2050): [path=/][flags=100352]
s3fs_readdir(2182): [path=/]
  list_bucket(2225): [path=/]
    ListBucketRequest(2270): [tpath=/]
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=1000
    RequestPerform(1483): HTTP response code 200
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
  readdir_multi_head(2105): [path=/][list=0]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478025][hit count=4]
    Request(3150): [count=0]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478047][hit count=5]
s3fs_getattr(691): [path=/sub1]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478047][hit count=6]
s3fs_getattr(691): [path=/]
s3fs_access(2646): [path=/sub1][mask=X_OK ]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478047][hit count=7]
s3fs_opendir(2050): [path=/sub1][flags=100352]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478048][hit count=8]
s3fs_readdir(2182): [path=/sub1]
    GetStat(170): stat cache hit [path=/sub1/][time=1422478049][hit count=9]
  list_bucket(2225): [path=/sub1]
    ListBucketRequest(2270): [tpath=/sub1]
    RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=sub1/&max-keys=1000
    RequestPerform(1483): HTTP response code 200
  readdir_multi_head(2105): [path=/sub1/][list=0]
    Request(3150): [count=2]
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 0 code: 28  msg: Timeout was reached), so retry this.
MultiRead(3113): failed to read(remaining: 1 code: 28  msg: Timeout was reached), so retry this.
multi_head_retry_callback(2079): Over retry count(5) limit(/sub1/sub2/).
$

$ ls -l
ls: cannot access s3: Transport endpoint is not connected
total 0
d????????? ? ?    ?    ?            ? s3
Originally created by @movence on GitHub (Jan 28, 2015). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/115 mounting a bucket using IAM written to /etc/passwd-s3fs `s3fs -d -f -o allow_other,use_cache=/tmp,retries=5,rw,multireq_max=7 test_bucket /mnt/s3` tried both with and without '-d -f' and got the similar problems The leaf files are in usually 3 or 4 deep down in directory hierarchy. ``` sub1 |-sub2 |-sub3 |-sub4 |-files... ``` reading files under /mnt/s3 is fine, but the connection gets lost even doing ls -l on sub3 level. It looks like the behavior is more random as going into sub directories. ``` $ sudo s3fs -d -f -o allow_other,use_cache=/tmp,retries=5,rw,multireq_max=7 test_bucket /mnt/s3 set_moutpoint_attribute(3291): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777) s3fs_init(2595): init s3fs_check_service(2894): check services. CheckBucket(2228): check a bucket. RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com/ RequestPerform(1483): HTTP response code 200 s3fs_getattr(691): [path=/] s3fs_getattr(691): [path=/] s3fs_access(2646): [path=/][mask=X_OK ] s3fs_opendir(2050): [path=/][flags=100352] s3fs_readdir(2182): [path=/] list_bucket(2225): [path=/] ListBucketRequest(2270): [tpath=/] RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=1000 RequestPerform(1483): HTTP response code 200 readdir_multi_head(2105): [path=/][list=0] Request(3150): [count=3] MultiRead(3113): failed to read(remaining: 2 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 2 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. AddStat(247): add stat cache entry[path=/sub1/] AddStat(247): add stat cache entry[path=/test1.txt] AddStat(247): add stat cache entry[path=/test.txt] GetStat(170): stat cache hit [path=/sub1/][time=1422478014][hit count=0] GetStat(170): stat cache hit [path=/test.txt][time=1422478014][hit count=0] GetStat(170): stat cache hit [path=/test1.txt][time=1422478014][hit count=0] s3fs_getattr(691): [path=/sub1] GetStat(170): stat cache hit [path=/sub1/][time=1422478014][hit count=1] s3fs_getattr(691): [path=/test.txt] GetStat(170): stat cache hit [path=/test.txt][time=1422478014][hit count=1] s3fs_getattr(691): [path=/test1.txt] GetStat(170): stat cache hit [path=/test1.txt][time=1422478014][hit count=1] s3fs_opendir(2050): [path=/][flags=100352] s3fs_readdir(2182): [path=/] list_bucket(2225): [path=/] ListBucketRequest(2270): [tpath=/] RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=1000 RequestPerform(1483): HTTP response code 200 readdir_multi_head(2105): [path=/][list=0] GetStat(170): stat cache hit [path=/sub1/][time=1422478014][hit count=2] GetStat(170): stat cache hit [path=/test.txt][time=1422478014][hit count=2] GetStat(170): stat cache hit [path=/test1.txt][time=1422478014][hit count=2] Request(3150): [count=0] GetStat(170): stat cache hit [path=/sub1/][time=1422478025][hit count=3] GetStat(170): stat cache hit [path=/test.txt][time=1422478025][hit count=3] GetStat(170): stat cache hit [path=/test1.txt][time=1422478025][hit count=3] s3fs_getattr(691): [path=/] s3fs_getattr(691): [path=/test1.txt] GetStat(170): stat cache hit [path=/test1.txt][time=1422478025][hit count=4] s3fs_access(2646): [path=/test1.txt][mask=W_OK ] GetStat(170): stat cache hit [path=/test1.txt][time=1422478025][hit count=5] s3fs_unlink(888): [path=/test1.txt] DeleteRequest(1787): [tpath=/test1.txt] RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com/test1.txt RequestPerform(1483): HTTP response code 204 DelStat(375): delete stat cache entry[path=/test1.txt] s3fs_getattr(691): [path=/test.txt] GetStat(170): stat cache hit [path=/test.txt][time=1422478025][hit count=4] s3fs_access(2646): [path=/test.txt][mask=W_OK ] GetStat(170): stat cache hit [path=/test.txt][time=1422478025][hit count=5] s3fs_unlink(888): [path=/test.txt] DeleteRequest(1787): [tpath=/test.txt] RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com/test.txt RequestPerform(1483): HTTP response code 204 DelStat(375): delete stat cache entry[path=/test.txt] s3fs_opendir(2050): [path=/][flags=100352] s3fs_readdir(2182): [path=/] list_bucket(2225): [path=/] ListBucketRequest(2270): [tpath=/] RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=&max-keys=1000 RequestPerform(1483): HTTP response code 200 append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty. readdir_multi_head(2105): [path=/][list=0] GetStat(170): stat cache hit [path=/sub1/][time=1422478025][hit count=4] Request(3150): [count=0] GetStat(170): stat cache hit [path=/sub1/][time=1422478047][hit count=5] s3fs_getattr(691): [path=/sub1] GetStat(170): stat cache hit [path=/sub1/][time=1422478047][hit count=6] s3fs_getattr(691): [path=/] s3fs_access(2646): [path=/sub1][mask=X_OK ] GetStat(170): stat cache hit [path=/sub1/][time=1422478047][hit count=7] s3fs_opendir(2050): [path=/sub1][flags=100352] GetStat(170): stat cache hit [path=/sub1/][time=1422478048][hit count=8] s3fs_readdir(2182): [path=/sub1] GetStat(170): stat cache hit [path=/sub1/][time=1422478049][hit count=9] list_bucket(2225): [path=/sub1] ListBucketRequest(2270): [tpath=/sub1] RequestPerform(1467): connecting to URL http://test_bucket.s3.amazonaws.com?delimiter=/&prefix=sub1/&max-keys=1000 RequestPerform(1483): HTTP response code 200 readdir_multi_head(2105): [path=/sub1/][list=0] Request(3150): [count=2] MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 0 code: 28 msg: Timeout was reached), so retry this. MultiRead(3113): failed to read(remaining: 1 code: 28 msg: Timeout was reached), so retry this. multi_head_retry_callback(2079): Over retry count(5) limit(/sub1/sub2/). $ ``` ``` $ ls -l ls: cannot access s3: Transport endpoint is not connected total 0 d????????? ? ? ? ? ? s3 ```
kerem closed this issue 2026-03-04 01:41:46 +03:00
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@movence commented on GitHub (Feb 4, 2015):

OK, deleting all existing directories helped it working normally. Creating multi-level directory hierarchy after starting from a scratch works fine.

Another issue that I am running into is that I have built the directory hierarchy on machine A, then mounted the same bucket on machine B. Machine B CANNOT even 'ls -l' on any of directories created by machine A through s3fs. I'm using s3fs on both machines, but B fails to read what A has created.
There is no issue reading files or directories created by machine B from A.

It does not look like this repository is "very" active, so I am just posting this while keeping my fingers crossed with NFS using EBS on EC2.

<!-- gh-comment-id:72783243 --> @movence commented on GitHub (Feb 4, 2015): OK, deleting all existing directories helped it working normally. Creating multi-level directory hierarchy after starting from a scratch works fine. Another issue that I am running into is that I have built the directory hierarchy on machine A, then mounted the same bucket on machine B. Machine B CANNOT even 'ls -l' on any of directories created by machine A through s3fs. I'm using s3fs on both machines, but B fails to read what A has created. There is no issue reading files or directories created by machine B from A. It does not look like this repository is "very" active, so I am just posting this while keeping my fingers crossed with NFS using EBS on EC2.
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Feb 4, 2015):

@movence s3fs is not well-suited for sharing files between multiple clients; it lacks the consistency and leasing protocols that NFS and CIFS provide. You might improve your experience by tuning the various cache and timeout values, e.g., stat_cache_expire. s3mper and YAS3FS might provide stronger guarantees via using some external storage.

<!-- gh-comment-id:72793819 --> @gaul commented on GitHub (Feb 4, 2015): @movence s3fs is not well-suited for sharing files between multiple clients; it lacks the consistency and leasing protocols that NFS and CIFS provide. You might improve your experience by tuning the various cache and timeout values, e.g., `stat_cache_expire`. [s3mper](https://github.com/Netflix/s3mper) and [YAS3FS](https://github.com/danilop/yas3fs) might provide stronger guarantees via using some external storage.
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@movence commented on GitHub (Feb 7, 2015):

@andrewgaul thanks! It looks like NFS does the job I was looking for.

<!-- gh-comment-id:73388726 --> @movence commented on GitHub (Feb 7, 2015): @andrewgaul thanks! It looks like NFS does the job I was looking for.
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 17, 2016):

I'm closing this issue, if you have a problem yet, please post new issue or reopen this issue.

Thanks in advance for your help.

<!-- gh-comment-id:172302229 --> @ggtakec commented on GitHub (Jan 17, 2016): I'm closing this issue, if you have a problem yet, please post new issue or reopen this issue. Thanks in advance for your help.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#70
No description provided.