starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2182] Cannot See Folders In The Bucket #1109

New issue
Open
opened 2026-03-04 01:51:27 +03:00 by kerem · 15 comments
kerem commented 2026-03-04 01:51:27 +03:00
Owner
Copy link

Originally created by @dnlchen on GitHub (Jun 6, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2182

Additional Information

Version of s3fs being used (s3fs --version)

V1.92

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.2

Kernel information (uname -r)

3.10.0-1160.90.1.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How to run s3fs, if applicable

[] command line
[x] /etc/fstab

log_bucket /logs fuse.s3fs _netdev,allow_other,umask=0002,noexec,endpoint=us-west-2,iam_role 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Jun 06 15:58:33 host-01 s3fs[1174]: s3fs version 1.92(unknown) : s3fs -o rw,noexec,allow_other,umask=0002,endpoint=us-west-2,iam_role,dev,suid log_bucket /logs
Jun 06 15:58:33 host-01 s3fs[1174]: Loaded mime information from /etc/mime.types
Jun 06 15:58:33 host-01 s3fs[1224]: s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function.
Jun 06 15:58:33 host-01 s3fs[1224]: init v1.92(commit:unknown) with OpenSSL, credential-library(built-in)
Jun 06 16:30:43 host-01 s3fs[29779]: s3fs version 1.92(unknown) : s3fs outputs
Jun 06 16:30:43 host-01 s3fs[29779]: Loaded mime information from /etc/mime.types
Jun 06 16:30:43 host-01 s3fs[29779]: s3fs: could not determine how to establish security credentials.

Details about issue

we can only see the files in the first level rather than any other folders in the bucket now.

Originally created by @dnlchen on GitHub (Jun 6, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2182 <!-- -------------------------------------------------------------------------- The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD. --------------------------------------------------------------------------- --> ### Additional Information #### Version of s3fs being used (`s3fs --version`) <!-- example: V1.91 (commit:b19262a) --> V1.92 #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) <!-- example: 2.9.2 --> 2.9.2 #### Kernel information (`uname -r`) <!-- example: 5.10.96-90.460.amzn2.x86_64 --> 3.10.0-1160.90.1.el7.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) <!-- command result --> NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" #### How to run s3fs, if applicable <!-- Describe the s3fs "command line" or "/etc/fstab" entry used. --> [] command line [x] /etc/fstab <!-- Executed command line or /etc/fastab entry --> ``` log_bucket /logs fuse.s3fs _netdev,allow_other,umask=0002,noexec,endpoint=us-west-2,iam_role 0 0 ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) <!-- if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages. --> ``` Jun 06 15:58:33 host-01 s3fs[1174]: s3fs version 1.92(unknown) : s3fs -o rw,noexec,allow_other,umask=0002,endpoint=us-west-2,iam_role,dev,suid log_bucket /logs Jun 06 15:58:33 host-01 s3fs[1174]: Loaded mime information from /etc/mime.types Jun 06 15:58:33 host-01 s3fs[1224]: s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function. Jun 06 15:58:33 host-01 s3fs[1224]: init v1.92(commit:unknown) with OpenSSL, credential-library(built-in) Jun 06 16:30:43 host-01 s3fs[29779]: s3fs version 1.92(unknown) : s3fs outputs Jun 06 16:30:43 host-01 s3fs[29779]: Loaded mime information from /etc/mime.types Jun 06 16:30:43 host-01 s3fs[29779]: s3fs: could not determine how to establish security credentials. ``` ### Details about issue <!-- Please describe the content of the issue in detail. --> we can only see the files in the first level rather than any other folders in the bucket now.
kerem commented 2026-03-04 01:51:28 +03:00
Author
Owner
Copy link

@dnlchen commented on GitHub (Jun 9, 2023):

should I add more options to show the folders?
/etc/fstab:
log_bucket /logs fuse.s3fs _netdev,allow_other,umask=0002,noexec,endpoint=us-west-2,iam_role 0 0

<!-- gh-comment-id:1584989070 --> @dnlchen commented on GitHub (Jun 9, 2023): should I add more options to show the folders? /etc/fstab: `log_bucket /logs fuse.s3fs _netdev,allow_other,umask=0002,noexec,endpoint=us-west-2,iam_role 0 0`
kerem commented 2026-03-04 01:51:28 +03:00
Author
Owner
Copy link

@dnlchen commented on GitHub (Jun 9, 2023):

I've downgraded s3fs-fuse to s3fs-fuse-1.91-4.el7.x86_64 to avoid the problem for now.

<!-- gh-comment-id:1585020280 --> @dnlchen commented on GitHub (Jun 9, 2023): I've downgraded s3fs-fuse to `s3fs-fuse-1.91-4.el7.x86_64` to avoid the problem for now.
kerem commented 2026-03-04 01:51:28 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 10, 2023):

@dnlchen
Are those folders created from a client other than s3fs?
If so, try starting it with compat_dir(and maybe complement_stat) option.

<!-- gh-comment-id:1585555488 --> @ggtakec commented on GitHub (Jun 10, 2023): @dnlchen Are those folders created from a client other than s3fs? If so, try starting it with `compat_dir`(and maybe `complement_stat`) option.
kerem commented 2026-03-04 01:51:28 +03:00
Author
Owner
Copy link

@iptizer commented on GitHub (Jun 19, 2023):

We have an issue that sound similar. Let me add from our side:

  • Downgrading to 1.91 is a workaround.
  • Adding compat_dir or complement_stat does NOT improve the situation.
  • Even though on linux level the missing directory is not shown => It is possible to cd directory (where directory is the name of the missing directory). Until the next remount, the directory is then visible.

=> We are creating "directories" with different tools than s3fs.

What debug logs would help to troubleshoot?

<!-- gh-comment-id:1597368064 --> @iptizer commented on GitHub (Jun 19, 2023): We have an issue that sound similar. Let me add from our side: * Downgrading to 1.91 is a workaround. * Adding `compat_dir` or `complement_stat` does NOT improve the situation. * Even though on linux level the missing directory is not shown => It is possible to `cd directory` (where directory is the name of the missing directory). Until the next remount, the directory is then visible. => We are creating "directories" with different tools than s3fs. What debug logs would help to troubleshoot?
kerem commented 2026-03-04 01:51:28 +03:00
Author
Owner
Copy link

@nobodyme commented on GitHub (Jun 21, 2023):

We have this exact same problem with the 1.92 version.
1.91 works fine as others as suggested as this continues to work on other systems.
and how did you manage to downgrade to s3fs-fuse-1.91-4.el7.x86_64 other than installing from source?
I couldn't find the RPM, if you did manage, please link it, thanks.

<!-- gh-comment-id:1600387733 --> @nobodyme commented on GitHub (Jun 21, 2023): We have this exact same problem with the 1.92 version. 1.91 works fine as others as suggested as this continues to work on other systems. and how did you manage to downgrade to `s3fs-fuse-1.91-4.el7.x86_64` other than installing from source? I couldn't find the RPM, if you did manage, please link it, thanks.
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@jacobtolar commented on GitHub (Jun 23, 2023):

@nobodyme , I found s3fs-fuse 1.91 downloadable from Fedora Koji

https://koji.fedoraproject.org/koji/packageinfo?packageID=27631
https://koji.fedoraproject.org/koji/buildinfo?buildID=1931091
https://kojipkgs.fedoraproject.org//packages/s3fs-fuse/1.91/1.el7/x86_64/s3fs-fuse-1.91-1.el7.x86_64.rpm

<!-- gh-comment-id:1605000648 --> @jacobtolar commented on GitHub (Jun 23, 2023): @nobodyme , I found s3fs-fuse 1.91 downloadable from Fedora Koji https://koji.fedoraproject.org/koji/packageinfo?packageID=27631 https://koji.fedoraproject.org/koji/buildinfo?buildID=1931091 https://kojipkgs.fedoraproject.org//packages/s3fs-fuse/1.91/1.el7/x86_64/s3fs-fuse-1.91-1.el7.x86_64.rpm
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@nobodyme commented on GitHub (Jun 23, 2023):

Awesome thanks

<!-- gh-comment-id:1605003936 --> @nobodyme commented on GitHub (Jun 23, 2023): Awesome thanks
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jun 25, 2023):

@ggtakec I can reproduce these symptoms on my bucket. I think our integration test might be incorrect. This seems like a major regression and we might need to run a 1.93 release immediately.

<!-- gh-comment-id:1606069947 --> @gaul commented on GitHub (Jun 25, 2023): @ggtakec I can reproduce these symptoms on my bucket. I think our integration test might be incorrect. This seems like a major regression and we might need to run a 1.93 release immediately.
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 25, 2023):

@dnlchen
I had overlooked the following line in your log.
I think you are failing to start the s3fs process.

Jun 06 16:30:43 host-01 s3fs[29779]: s3fs: could not determine how to establish security credentials.

I think it's probably due to the credential issue for an authentication.
I would like to determine the situation in detail, so could you start with dbglevel=info and let me know the s3fs log?

@iptizer @nobodyme @jacobtolar
We do not recommend downgrading.
I would like to know more about your situation, so could you tell me the log(start with dbglevel=info) when you start it?
(Please post another issue if you think it has a different cause than this issue.)

Thanks in advance for your assistance.

<!-- gh-comment-id:1606070287 --> @ggtakec commented on GitHub (Jun 25, 2023): @dnlchen I had overlooked the following line in your log. I think you are failing to start the s3fs process. ``` Jun 06 16:30:43 host-01 s3fs[29779]: s3fs: could not determine how to establish security credentials. ``` I think it's probably due to the credential issue for an authentication. I would like to determine the situation in detail, so could you start with `dbglevel=info` and let me know the s3fs log? @iptizer @nobodyme @jacobtolar We do not recommend downgrading. I would like to know more about your situation, so could you tell me the log(start with dbglevel=info) when you start it? (Please post another issue if you think it has a different cause than this issue.) Thanks in advance for your assistance.
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 25, 2023):

@gaul If s3fs can be started, the problem may be caused by modifying the compat_dir feature.
(However, if you check the first log shown in this issue, s3fs has not started due to authentication issues)

<!-- gh-comment-id:1606071512 --> @ggtakec commented on GitHub (Jun 25, 2023): @gaul If s3fs can be started, the problem may be caused by modifying the `compat_dir` feature. (However, if you check the first log shown in this issue, s3fs has not started due to authentication issues)
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 25, 2023):

@iptizer @nobodyme @jacobtolar @gaul
#2190 seems to be the same phenomenon.

<!-- gh-comment-id:1606082403 --> @ggtakec commented on GitHub (Jun 25, 2023): @iptizer @nobodyme @jacobtolar @gaul #2190 seems to be the same phenomenon.
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@dnlchen commented on GitHub (Jun 27, 2023):

@dnlchen I had overlooked the following line in your log. I think you are failing to start the s3fs process.

Jun 06 16:30:43 host-01 s3fs[29779]: s3fs: could not determine how to establish security credentials.

I think it's probably due to the credential issue for an authentication. I would like to determine the situation in detail, so could you start with dbglevel=info and let me know the s3fs log?

@iptizer @nobodyme @jacobtolar We do not recommend downgrading. I would like to know more about your situation, so could you tell me the log(start with dbglevel=info) when you start it? (Please post another issue if you think it has a different cause than this issue.)

Thanks in advance for your assistance.

I think that the credential is correct because we've been using the the same IAM role for years and it's working with 1.91. We can see the files in the mount, just cannot see those folders.

<!-- gh-comment-id:1609991152 --> @dnlchen commented on GitHub (Jun 27, 2023): > @dnlchen I had overlooked the following line in your log. I think you are failing to start the s3fs process. > > ``` > Jun 06 16:30:43 host-01 s3fs[29779]: s3fs: could not determine how to establish security credentials. > ``` > > I think it's probably due to the credential issue for an authentication. I would like to determine the situation in detail, so could you start with `dbglevel=info` and let me know the s3fs log? > > @iptizer @nobodyme @jacobtolar We do not recommend downgrading. I would like to know more about your situation, so could you tell me the log(start with dbglevel=info) when you start it? (Please post another issue if you think it has a different cause than this issue.) > > Thanks in advance for your assistance. I think that the credential is correct because we've been using the the same IAM role for years and it's working with 1.91. We can see the files in the mount, just cannot see those folders.
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jul 2, 2023):

@iptizer @dnlchen @nobodyme @jacobtolar
I merged PR #2204.
I want to check if the cause is the same, so if you can, could you try the latest code(current master branch)?

<!-- gh-comment-id:1616272067 --> @ggtakec commented on GitHub (Jul 2, 2023): @iptizer @dnlchen @nobodyme @jacobtolar I merged PR #2204. I want to check if the cause is the same, so if you can, could you try the latest code(current master branch)?
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Sep 8, 2023):

Please test with 1.93 which fixes the directory regression.

<!-- gh-comment-id:1710950874 --> @gaul commented on GitHub (Sep 8, 2023): Please test with 1.93 which fixes the directory regression.
kerem commented 2026-03-04 01:51:29 +03:00
Author
Owner
Copy link

@Leonski13 commented on GitHub (Nov 30, 2023):

We are experiencing the same problem that files and folders in a bucket are not shown. The problem occurs only with some folders, not all folders are affected. I have tested with 1.93 and 1.92. The issue does not appear in 1.91

<!-- gh-comment-id:1833380818 --> @Leonski13 commented on GitHub (Nov 30, 2023): We are experiencing the same problem that files and folders in a bucket are not shown. The problem occurs only with some folders, not all folders are affected. I have tested with `1.93` and `1.92`. The issue does not appear in `1.91`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1109
No description provided.