starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 05:16:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2187] dir on windows does not list anything #1111

New issue
Open
opened 2026-03-04 01:51:29 +03:00 by kerem · 6 comments
kerem commented 2026-03-04 01:51:29 +03:00
Owner
Copy link

Originally created by @jsm222 on GitHub (Jun 11, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2187

Version of s3fs being used (s3fs --version)

V1.92 (commit:580775b) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

fuse for windows
see https://gist.github.com/jsm222/2683428cb6f84f22ea4127c28fcfb12e
This new gist revision seems more stable on windows.

Originally created by @jsm222 on GitHub (Jun 11, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2187 #### Version of s3fs being used (`s3fs --version`) V1.92 (commit:580775b) with OpenSSL #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) fuse for windows see https://gist.github.com/jsm222/2683428cb6f84f22ea4127c28fcfb12e This new gist revision seems more stable on windows.
kerem commented 2026-03-04 01:51:30 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 12, 2023):

@jsm222
Have you tried the compat_dir (and complement_stat) options?
After v1.91, the default compatibility for folders(objects) created from other s3fs clients has changed.
If the directory you can't see was created by a client other than s3fs, you must give the compat_dir option.

<!-- gh-comment-id:1587152764 --> @ggtakec commented on GitHub (Jun 12, 2023): @jsm222 Have you tried the `compat_dir` (and `complement_stat`) options? After v1.91, the default compatibility for folders(objects) created from other s3fs clients has changed. If the directory you can't see was created by a client other than s3fs, you must give the `compat_dir` option.
kerem commented 2026-03-04 01:51:30 +03:00
Author
Owner
Copy link

@jsm222 commented on GitHub (Jun 12, 2023):

same result. Without the patch against curl_multi.cpp in the gist I get file not found with one of or both -o compat_dir and complement_stat

2023-06-12T14:58:59.645Z [CURL DBG] * Connection #0 to host s3.eu-central-1.amazonaws.com left intact
2023-06-12T14:58:59.645Z [INF]       curl.cpp:RequestPerform(2519): HTTP response code 200
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2023-06-12T14:58:59.645Z [WAN] s3fs.cpp:readdir_multi_head(3248): error occurred in multi request(errno=-5), but continue..
<!-- gh-comment-id:1587536358 --> @jsm222 commented on GitHub (Jun 12, 2023): same result. Without the patch against curl_multi.cpp in the gist I get file not found with one of or both -o compat_dir and complement_stat ``` 2023-06-12T14:58:59.645Z [CURL DBG] * Connection #0 to host s3.eu-central-1.amazonaws.com left intact 2023-06-12T14:58:59.645Z [INF] curl.cpp:RequestPerform(2519): HTTP response code 200 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2023-06-12T14:58:59.645Z [WAN] s3fs.cpp:readdir_multi_head(3248): error occurred in multi request(errno=-5), but continue.. ```
kerem commented 2026-03-04 01:51:30 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Sep 8, 2023):

Could you test on 1.93 which fixes the directory regression?

<!-- gh-comment-id:1710950116 --> @gaul commented on GitHub (Sep 8, 2023): Could you test on 1.93 which fixes the directory regression?
kerem commented 2026-03-04 01:51:30 +03:00
Author
Owner
Copy link

@stevenlafl commented on GitHub (Feb 7, 2024):

I have tested it on 1.93 on windows. There has been no change. I am using:

.\s3fs somebucketname X: -o passwd_file=C:\\Users\\Administrator\\.passwd-s3fs -o dbglevel=info -f -o curldbg -o umask=000 -o compat_dir=1 -o complement_stat=1

I did upload the file that exists there using S3FS. I don't see files regardless of the method I use to upload them.

I DO however see directories.

Here's a gist of the logs:
https://gist.github.com/stevenlafl/d12d131b9db6b064e8648492677a15ce

I can tell s3fs-fuse -sees- the files, because:

2024-02-07T01:17:38.764Z [INF]       curl.cpp:PreHeadRequest(3283): [tpath=/mime.types][bpath=/mime.types][save=/mime.types][sseckeypos=xxxxxxxxxxxxxxxxxxxxx]
...
2024-02-07T01:17:38.768Z [INF]       curl.cpp:insertV4Headers(2916): computing signature [HEAD] [/ftproot/mime.types] [] []

These lines show the file I have uploaded there, /ftproot/mime.types

When I add, lol.xml and refresh the file listing, it's still empty in File Explorer but the logs show:

2024-02-07T01:27:54.932Z [INF] s3fs.cpp:s3fs_opendir(3139): [path=/][flags=0x0]
...
2024-02-07T01:27:55.009Z [INF]   s3fs.cpp:readdir_multi_head(3251): [path=/][list=0]
2024-02-07T01:27:55.010Z [INF]       curl.cpp:PreHeadRequest(3283): [tpath=/lol.xml][bpath=/lol.xml][save=/lol.xml][sseckeypos=xxxxxxxxx]
2024-02-07T01:27:55.011Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3-us-east-2.amazonaws.com/somebucketname/ftproot/lol.xml
2024-02-07T01:27:55.011Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://somebucketname.s3-us-east-2.amazonaws.com/ftproot/lol.xml
2024-02-07T01:27:55.012Z [INF]       curl.cpp:PreHeadRequest(3283): [tpath=/mime.types][bpath=/mime.types][save=/mime.types][sseckeypos=xxxxxxxxx]
2024-02-07T01:27:55.012Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3-us-east-2.amazonaws.com/somebucketname/ftproot/mime.types
2024-02-07T01:27:55.013Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://somebucketname.s3-us-east-2.amazonaws.com/ftproot/mime.types
2024-02-07T01:27:55.014Z [INF]       curl_multi.cpp:Request(324): [count=2]
2024-02-07T01:27:55.049Z [INF]       curl.cpp:insertV4Headers(2916): computing signature [HEAD] [/ftproot/lol.xml] [] []
2024-02-07T01:27:55.067Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3-us-east-2.amazonaws.com
2024-02-07T01:27:55.068Z [INF]       curl.cpp:insertV4Headers(2916): computing signature [HEAD] [/ftproot/mime.types] [] []
2024-02-07T01:27:55.069Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3-us-east-2.amazonaws.com
2024-02-07T01:27:55.114Z [INF]       curl.cpp:RequestPerform(2544): HTTP response code 200
2024-02-07T01:27:55.139Z [INF]       curl.cpp:RequestPerform(2544): HTTP response code 200
2024-02-07T01:27:55.139Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2024-02-07T01:27:55.140Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22)
2024-02-07T01:27:55.140Z [WAN] s3fs.cpp:readdir_multi_head(3321): error occurred in multi request(errno=-5), but continue...
<!-- gh-comment-id:1931093833 --> @stevenlafl commented on GitHub (Feb 7, 2024): I have tested it on 1.93 on windows. There has been no change. I am using: ``` .\s3fs somebucketname X: -o passwd_file=C:\\Users\\Administrator\\.passwd-s3fs -o dbglevel=info -f -o curldbg -o umask=000 -o compat_dir=1 -o complement_stat=1 ``` I did upload the file that exists there using S3FS. I don't see files regardless of the method I use to upload them. **I *DO* however see directories.** Here's a gist of the logs: https://gist.github.com/stevenlafl/d12d131b9db6b064e8648492677a15ce I can tell `s3fs-fuse` -sees- the files, because: ``` 2024-02-07T01:17:38.764Z [INF] curl.cpp:PreHeadRequest(3283): [tpath=/mime.types][bpath=/mime.types][save=/mime.types][sseckeypos=xxxxxxxxxxxxxxxxxxxxx] ... 2024-02-07T01:17:38.768Z [INF] curl.cpp:insertV4Headers(2916): computing signature [HEAD] [/ftproot/mime.types] [] [] ``` These lines show the file I have uploaded there, `/ftproot/mime.types` When I add, `lol.xml` and refresh the file listing, it's still empty in File Explorer but the logs show: ``` 2024-02-07T01:27:54.932Z [INF] s3fs.cpp:s3fs_opendir(3139): [path=/][flags=0x0] ... 2024-02-07T01:27:55.009Z [INF] s3fs.cpp:readdir_multi_head(3251): [path=/][list=0] 2024-02-07T01:27:55.010Z [INF] curl.cpp:PreHeadRequest(3283): [tpath=/lol.xml][bpath=/lol.xml][save=/lol.xml][sseckeypos=xxxxxxxxx] 2024-02-07T01:27:55.011Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3-us-east-2.amazonaws.com/somebucketname/ftproot/lol.xml 2024-02-07T01:27:55.011Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://somebucketname.s3-us-east-2.amazonaws.com/ftproot/lol.xml 2024-02-07T01:27:55.012Z [INF] curl.cpp:PreHeadRequest(3283): [tpath=/mime.types][bpath=/mime.types][save=/mime.types][sseckeypos=xxxxxxxxx] 2024-02-07T01:27:55.012Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3-us-east-2.amazonaws.com/somebucketname/ftproot/mime.types 2024-02-07T01:27:55.013Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://somebucketname.s3-us-east-2.amazonaws.com/ftproot/mime.types 2024-02-07T01:27:55.014Z [INF] curl_multi.cpp:Request(324): [count=2] 2024-02-07T01:27:55.049Z [INF] curl.cpp:insertV4Headers(2916): computing signature [HEAD] [/ftproot/lol.xml] [] [] 2024-02-07T01:27:55.067Z [INF] curl_util.cpp:url_to_host(334): url is https://s3-us-east-2.amazonaws.com 2024-02-07T01:27:55.068Z [INF] curl.cpp:insertV4Headers(2916): computing signature [HEAD] [/ftproot/mime.types] [] [] 2024-02-07T01:27:55.069Z [INF] curl_util.cpp:url_to_host(334): url is https://s3-us-east-2.amazonaws.com 2024-02-07T01:27:55.114Z [INF] curl.cpp:RequestPerform(2544): HTTP response code 200 2024-02-07T01:27:55.139Z [INF] curl.cpp:RequestPerform(2544): HTTP response code 200 2024-02-07T01:27:55.139Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2024-02-07T01:27:55.140Z [ERR] curl_multi.cpp:MultiPerform(191): failed pthread_join - rc(22) 2024-02-07T01:27:55.140Z [WAN] s3fs.cpp:readdir_multi_head(3321): error occurred in multi request(errno=-5), but continue... ```
kerem commented 2026-03-04 01:51:30 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 12, 2024):

@stevenlafl Thanks for some log.
I checked your 2'nd log and I have a rough idea of what's going on.
At first, s3fs listed file objects and detecting two objects.
After that, s3fs tried to get the details(status) of each file and these requests also received responses.
However, for some reason, an error occurs during s3fs processing (it seems that the argument passed to the pthread_join function(thread ID incorrect)).

From your logs, I can only understand this situation, but I have not yet determined the cause.
If possible, I think it would be helpful to have more detailed logs for analysis.

<!-- gh-comment-id:1938126568 --> @ggtakec commented on GitHub (Feb 12, 2024): @stevenlafl Thanks for some log. I checked your 2'nd log and I have a rough idea of what's going on. At first, s3fs listed file objects and detecting two objects. After that, s3fs tried to get the details(status) of each file and these requests also received responses. However, for some reason, an error occurs during s3fs processing (it seems that the argument passed to the pthread_join function(thread ID incorrect)). From your logs, I can only understand this situation, but I have not yet determined the cause. If possible, I think it would be helpful to have more detailed logs for analysis.
kerem commented 2026-03-04 01:51:30 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 12, 2024):

If the handle passed to pthread_join is unique, this error should not occur, but there may be a problem somewhere.(I haven't been able to find it yet.)
The Stat cache may send a HEAD request again if it is cached out or if it is different from when it was cached.
Thanks in advance for you help.

<!-- gh-comment-id:1938145853 --> @ggtakec commented on GitHub (Feb 12, 2024): If the handle passed to pthread_join is unique, this error should not occur, but there may be a problem somewhere.(I haven't been able to find it yet.) The Stat cache may send a HEAD request again if it is cached out or if it is different from when it was cached. Thanks in advance for you help.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1111
No description provided.