[GH-ISSUE #2041] s3fs continues to not mount reliably #1029

New issue

Closed

opened 2026-03-04 01:50:47 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:50:47 +03:00

Owner

Originally created by @cyb3rz3us on GitHub (Sep 28, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2041

Version of s3fs being used (s3fs --version)

s3fs-fuse.x86_64 1.91-1.el8 @epel

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

fuse.x86_64                         2.9.7-15.el8                            @baseos
fuse-common.x86_64                  3.3.0-15.el8                            @baseos
fuse-libs.x86_64                    2.9.7-15.el8                            @anaconda

Kernel information (uname -r)

4.18.0-372.9.1.el8.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Rocky Linux"
VERSION="8.6 (Green Obsidian)"

s3fs command line used, if applicable

The following commands were attempted twice each but using different authentication methods as follows: (a) the ~/.aws/credentials file; or (b) the -o passwd_file option.
Both authentication options contains the same user credentials.
The user is the same as is currently used for all other AWS CLI S3 operations which all work without issue.
The group containing this user has AmazonS3FullAccess permissions.

s3fs -d [ BUCKET_NAME ] /s3-test -o url=https://s3-us-west-2.amazonaws.com
s3fs -d [ BUCKET_NAME ] /s3-test -o url="https://s3-us-west-2.amazonaws.com"
s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2"
s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2" -o url=https://s3-us-west-2.amazonaws.com
s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2" -o url="https://s3-us-west-2.amazonaws.com"

/etc/fstab entry, if applicable

Not applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

The following is from s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2" -o passwd_file=~/passwd-s3fs.TMP but the other CLI attempts result in nearly the exact same debug message:

Sep 28 16:07:57 test1 s3fs[31352]: s3fs version 1.91(unknown) : s3fs -d -o endpoint=us-west-2 -o passwd_file=/home/[ REDACTED ]/passwd-s3fs.TMP [ BUCKET_NAME ] /s3-test
Sep 28 16:07:57 test1 s3fs[31352]: s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF]
Sep 28 16:07:57 test1 s3fs[31352]:    PROC(uid=1001, gid=1001) - MountPoint(uid=1001, gid=0, mode=40700)
Sep 28 16:07:57 test1 s3fs[31352]: Loaded mime information from /etc/mime.types
Sep 28 16:07:57 test1 s3fs[31352]: The path to cache top dir is empty, thus not need to check permission.
Sep 28 16:07:57 test1 s3fs[31355]: init v1.91(commit:unknown) with OpenSSL
Sep 28 16:07:57 test1 s3fs[31355]: check services.
Sep 28 16:07:57 test1 s3fs[31355]:      check a bucket.
Sep 28 16:07:57 test1 s3fs[31355]:      URL is https://s3.amazonaws.com/[ BUCKET_NAME ]/
Sep 28 16:07:57 test1 s3fs[31355]:      URL changed is https://[ BUCKET_NAME ].s3.amazonaws.com/
Sep 28 16:07:57 test1 s3fs[31355]:      computing signature [GET] [/] [] []
Sep 28 16:07:57 test1 s3fs[31355]:      url is https://s3.amazonaws.com
Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:RequestPerform(2371): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EMXX26A82TP4DH</RequestId><HostId>jYt6kH6ugLw3yQqi53sYwsuUL1Pt+gZcMfaJhj++HOUxM0pkGXh3PwCEHKAV4vqSwrGtoHjZP7w=</HostId></Error>
Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:CheckBucket(3515): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EMXX26A82TP4DH</RequestId><HostId>jYt6kH6ugLw3yQqi53sYwsuUL1Pt+gZcMfaJhj++HOUxM0pkGXh3PwCEHKAV4vqSwrGtoHjZP7w=</HostId></Error>
Sep 28 16:07:57 test1 s3fs[31355]: s3fs.cpp:s3fs_check_service(3572): Failed to connect by sigv4, so retry to connect by signature version 2.
Sep 28 16:07:57 test1 s3fs[31355]: Pool full: destroy the oldest handler
Sep 28 16:07:57 test1 s3fs[31355]:      check a bucket.
Sep 28 16:07:57 test1 s3fs[31355]:      URL is https://s3.amazonaws.com/[ BUCKET_NAME ]/
Sep 28 16:07:57 test1 s3fs[31355]:      URL changed is https://[ BUCKET_NAME ].s3.amazonaws.com/
Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:RequestPerform(2371): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EQ946TMMF8QFGN</RequestId><HostId>JCx+4CutZQrXzsX8/C4sLihfsXzADpGMjo5L3Qfihy0VzdTxqhq4+6moIazUmPLhF5SJ7kpZDzQ=</HostId></Error>
Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:CheckBucket(3515): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EQ946TMMF8QFGN</RequestId><HostId>JCx+4CutZQrXzsX8/C4sLihfsXzADpGMjo5L3Qfihy0VzdTxqhq4+6moIazUmPLhF5SJ7kpZDzQ=</HostId></Error>
Sep 28 16:07:57 test1 s3fs[31355]: s3fs.cpp:s3fs_check_service(3587): invalid credentials(host=https://s3.amazonaws.com) - result of checking service.
Sep 28 16:07:57 test1 s3fs[31355]: s3fs.cpp:s3fs_exit_fuseloop(3383): Exiting FUSE event loop due to errors
Sep 28 16:07:57 test1 systemd[1]: s3\x2dtest.mount: Succeeded.
Sep 28 16:07:57 test1 s3fs[31355]: destroy

Details about issue

I have buckets is several regions. Oddly, I have no problem connecting to a bucket in us-east-1; the issue only exists for buckets in all other regions despite using the -o endpoint and\or -o url options.
Again, I can use the aws s3 command all day on the same set of buckets using the exact same user credentials without any issue.

Originally created by @cyb3rz3us on GitHub (Sep 28, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2041 #### Version of s3fs being used (s3fs --version) `s3fs-fuse.x86_64 1.91-1.el8 @epel` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` fuse.x86_64 2.9.7-15.el8 @baseos fuse-common.x86_64 3.3.0-15.el8 @baseos fuse-libs.x86_64 2.9.7-15.el8 @anaconda ``` #### Kernel information (uname -r) 4.18.0-372.9.1.el8.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Rocky Linux" VERSION="8.6 (Green Obsidian)" #### s3fs command line used, if applicable * The following commands were attempted twice each but using different authentication methods as follows: **(a)** the `~/.aws/credentials` file; or **(b)** the `-o passwd_file` option. * Both authentication options contains the same user credentials. * The user is the same as is currently used for all other AWS CLI S3 operations which all work without issue. * The group containing this user has `AmazonS3FullAccess` permissions. ```s3fs -d [ BUCKET_NAME ] /s3-test -o url=https://s3-us-west-2.amazonaws.com``` ```s3fs -d [ BUCKET_NAME ] /s3-test -o url="https://s3-us-west-2.amazonaws.com"``` ```s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2"``` ```s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2" -o url=https://s3-us-west-2.amazonaws.com``` ```s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2" -o url="https://s3-us-west-2.amazonaws.com"``` #### /etc/fstab entry, if applicable Not applicable #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) * The following is from `s3fs -d [ BUCKET_NAME ] /s3-test -o endpoint="us-west-2" -o passwd_file=~/passwd-s3fs.TMP` but the other CLI attempts result in nearly the exact same debug message: ``` Sep 28 16:07:57 test1 s3fs[31352]: s3fs version 1.91(unknown) : s3fs -d -o endpoint=us-west-2 -o passwd_file=/home/[ REDACTED ]/passwd-s3fs.TMP [ BUCKET_NAME ] /s3-test Sep 28 16:07:57 test1 s3fs[31352]: s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF] Sep 28 16:07:57 test1 s3fs[31352]: PROC(uid=1001, gid=1001) - MountPoint(uid=1001, gid=0, mode=40700) Sep 28 16:07:57 test1 s3fs[31352]: Loaded mime information from /etc/mime.types Sep 28 16:07:57 test1 s3fs[31352]: The path to cache top dir is empty, thus not need to check permission. Sep 28 16:07:57 test1 s3fs[31355]: init v1.91(commit:unknown) with OpenSSL Sep 28 16:07:57 test1 s3fs[31355]: check services. Sep 28 16:07:57 test1 s3fs[31355]: check a bucket. Sep 28 16:07:57 test1 s3fs[31355]: URL is https://s3.amazonaws.com/[ BUCKET_NAME ]/ Sep 28 16:07:57 test1 s3fs[31355]: URL changed is https://[ BUCKET_NAME ].s3.amazonaws.com/ Sep 28 16:07:57 test1 s3fs[31355]: computing signature [GET] [/] [] [] Sep 28 16:07:57 test1 s3fs[31355]: url is https://s3.amazonaws.com Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:RequestPerform(2371): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EMXX26A82TP4DH</RequestId><HostId>jYt6kH6ugLw3yQqi53sYwsuUL1Pt+gZcMfaJhj++HOUxM0pkGXh3PwCEHKAV4vqSwrGtoHjZP7w=</HostId></Error> Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:CheckBucket(3515): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EMXX26A82TP4DH</RequestId><HostId>jYt6kH6ugLw3yQqi53sYwsuUL1Pt+gZcMfaJhj++HOUxM0pkGXh3PwCEHKAV4vqSwrGtoHjZP7w=</HostId></Error> Sep 28 16:07:57 test1 s3fs[31355]: s3fs.cpp:s3fs_check_service(3572): Failed to connect by sigv4, so retry to connect by signature version 2. Sep 28 16:07:57 test1 s3fs[31355]: Pool full: destroy the oldest handler Sep 28 16:07:57 test1 s3fs[31355]: check a bucket. Sep 28 16:07:57 test1 s3fs[31355]: URL is https://s3.amazonaws.com/[ BUCKET_NAME ]/ Sep 28 16:07:57 test1 s3fs[31355]: URL changed is https://[ BUCKET_NAME ].s3.amazonaws.com/ Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:RequestPerform(2371): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EQ946TMMF8QFGN</RequestId><HostId>JCx+4CutZQrXzsX8/C4sLihfsXzADpGMjo5L3Qfihy0VzdTxqhq4+6moIazUmPLhF5SJ7kpZDzQ=</HostId></Error> Sep 28 16:07:57 test1 s3fs[31355]: curl.cpp:CheckBucket(3515): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>29EQ946TMMF8QFGN</RequestId><HostId>JCx+4CutZQrXzsX8/C4sLihfsXzADpGMjo5L3Qfihy0VzdTxqhq4+6moIazUmPLhF5SJ7kpZDzQ=</HostId></Error> Sep 28 16:07:57 test1 s3fs[31355]: s3fs.cpp:s3fs_check_service(3587): invalid credentials(host=https://s3.amazonaws.com) - result of checking service. Sep 28 16:07:57 test1 s3fs[31355]: s3fs.cpp:s3fs_exit_fuseloop(3383): Exiting FUSE event loop due to errors Sep 28 16:07:57 test1 systemd[1]: s3\x2dtest.mount: Succeeded. Sep 28 16:07:57 test1 s3fs[31355]: destroy ``` ### Details about issue * I have buckets is several regions. Oddly, I have no problem connecting to a bucket in `us-east-1`; the issue only exists for buckets in all other regions despite using the `-o endpoint` and\or `-o url` options. * Again, I can use the `aws s3` command all day on the same set of buckets using the exact same user credentials without any issue.

kerem closed this issue

2026-03-04 01:50:47 +03:00

kerem commented

2026-03-04 01:50:48 +03:00

Author

Owner

@cyb3rz3us commented on GitHub (Sep 29, 2022):

This issue has been resolved and the root cause is very corner case. However, I am still going to post what happened to provide a possible clue as something to watch out for in your AWS implementation.

We had an old policy in place that limited which buckets could be accessed via a given S3 endpoint. This had been implemented in such a way that only a very small subset of connection attempts would experience this issue.

So for me, the takeaway is this: When in doubt - always double, triple, quadruple check your permissions and polices.

@cyb3rz3us commented on GitHub (Sep 29, 2022): **This issue has been resolved** and the root cause is very corner case. However, I am still going to post what happened to provide a possible clue as something to watch out for in your AWS implementation. We had an old policy in place that limited which buckets could be accessed via a given S3 endpoint. This had been implemented in such a way that only a very small subset of connection attempts would experience this issue. So for me, the takeaway is this: When in doubt - **always double, triple, quadruple check your permissions and polices**.