[GH-ISSUE #1535] s3fs can't connect using MFA and role setup from command #805

Open
opened 2026-03-04 01:48:56 +03:00 by kerem · 10 comments
Owner

Originally created by @emper0r on GitHub (Jan 25, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1535

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.88 (commit:6d65e30) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

s3fs-fuse-1.87-1.el7.x86_64
fuse-libs-2.9.2-11.el7.x86_64

Kernel information (uname -r)

3.10.0-1160.6.1.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

s3fs command line used, if applicable

using /etc/passwd-s3fs

s3fs -o profile="devops" -o passwd_file=/etc/passwd-s3fs bucket:/upload-test /mnt/ -o url=https://s3.amazonaws.com/ -o allow_other -o dbglevel=info -f -o curldbg

[CURL DBG] * Found bundle for host bucket.s3.amazonaws.com: 0x7f830c008730 [serially]
[CURL DBG] * Can not multiplex, even if we wanted to!
[CURL DBG] * Re-using existing connection! (#0) with host bucket.s3.amazonaws.com

[CURL DBG] * Connected to bucket.s3.amazonaws.com (52.217.105.164) port 443 (#0)

[CURL DBG] > GET /upload-test/ HTTP/1.1
[CURL DBG] > Host: bucket.s3.amazonaws.com
[CURL DBG] > User-Agent: s3fs/1.88 (commit hash 6d65e30; OpenSSL)
[CURL DBG] > Accept: */*
[CURL DBG] > Date: Mon, 25 Jan 2021 13:16:25 GMT
[CURL DBG] >
[CURL DBG] * Mark bundle as not supporting multiuse
[CURL DBG] < HTTP/1.1 403 Forbidden
[CURL DBG] < x-amz-request-id: 2A9E8277
[CURL DBG] < x-amz-id-2: YuudBffu8H+tjN0+prNSx9cx83yLpZqWs=
[CURL DBG] < Content-Type: application/xml
[CURL DBG] < Transfer-Encoding: chunked
[CURL DBG] < Date: Mon, 25 Jan 2021 13:16:25 GMT
[CURL DBG] < Server: AmazonS3

not using /etc/passwd-s3fs but using assuming role

s3fs -o profile="devops" -o iam_role="devops_test" bucket:/upload-test /mnt/ -o url=https://s3.amazonaws.com/ -o allow_other -o dbglevel=info -f -o curldbg

[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from [CRT] to [INF]
[INF]     s3fs.cpp:set_mountpoint_attribute(3993): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[INF] curl.cpp:InitMimeType(438): Loaded mime information from /etc/mime.types
[INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission.
[INF] s3fs.cpp:s3fs_init(3304): init v1.88(commit:6d65e30) with OpenSSL
[INF] s3fs.cpp:s3fs_check_service(3420): check services.
[INF] curl.cpp:CheckIAMCredentialUpdate(1721): IAM Access Token refreshing...
[INF]       curl.cpp:GetIAMCredentials(2742): [IAM role=devops_test]

**[CURL DBG] *   Trying 169.254.169.254:80...   <--- that IP can't be.**

/etc/fstab entry, if applicable

- 1st attempt using file not show mount point
- 2nd attempt  using role then can see but still blocked because command still hang-up using a wrong s3 public IP address
s3fs on /mnt type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

2nd attempt using assuming role (because use MFA) remains hung-up

Details about issue

it's strange using correct role, seems like is connecting but use a wrong IP address public for s3 and hung up without respond.
and checking by shell using aws cli works using credentials with mfa obviusly

aws s3 ls --region eu-south-1 --profile devops s3://upload-test
2021-01-23 13:42:32          0 s3-test.txt
Originally created by @emper0r on GitHub (Jan 25, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1535 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) ```Amazon Simple Storage Service File System V1.88 (commit:6d65e30) with OpenSSL``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ```fuse3-libs-3.6.1-4.el7.x86_64 s3fs-fuse-1.87-1.el7.x86_64 fuse-libs-2.9.2-11.el7.x86_64 ``` #### Kernel information (uname -r) ```3.10.0-1160.6.1.el7.x86_64``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ```NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" ``` #### s3fs command line used, if applicable using /etc/passwd-s3fs ``` s3fs -o profile="devops" -o passwd_file=/etc/passwd-s3fs bucket:/upload-test /mnt/ -o url=https://s3.amazonaws.com/ -o allow_other -o dbglevel=info -f -o curldbg [CURL DBG] * Found bundle for host bucket.s3.amazonaws.com: 0x7f830c008730 [serially] [CURL DBG] * Can not multiplex, even if we wanted to! [CURL DBG] * Re-using existing connection! (#0) with host bucket.s3.amazonaws.com [CURL DBG] * Connected to bucket.s3.amazonaws.com (52.217.105.164) port 443 (#0) [CURL DBG] > GET /upload-test/ HTTP/1.1 [CURL DBG] > Host: bucket.s3.amazonaws.com [CURL DBG] > User-Agent: s3fs/1.88 (commit hash 6d65e30; OpenSSL) [CURL DBG] > Accept: */* [CURL DBG] > Date: Mon, 25 Jan 2021 13:16:25 GMT [CURL DBG] > [CURL DBG] * Mark bundle as not supporting multiuse [CURL DBG] < HTTP/1.1 403 Forbidden [CURL DBG] < x-amz-request-id: 2A9E8277 [CURL DBG] < x-amz-id-2: YuudBffu8H+tjN0+prNSx9cx83yLpZqWs= [CURL DBG] < Content-Type: application/xml [CURL DBG] < Transfer-Encoding: chunked [CURL DBG] < Date: Mon, 25 Jan 2021 13:16:25 GMT [CURL DBG] < Server: AmazonS3 ``` not using /etc/passwd-s3fs but using assuming role ``` s3fs -o profile="devops" -o iam_role="devops_test" bucket:/upload-test /mnt/ -o url=https://s3.amazonaws.com/ -o allow_other -o dbglevel=info -f -o curldbg [CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(3993): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [INF] curl.cpp:InitMimeType(438): Loaded mime information from /etc/mime.types [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission. [INF] s3fs.cpp:s3fs_init(3304): init v1.88(commit:6d65e30) with OpenSSL [INF] s3fs.cpp:s3fs_check_service(3420): check services. [INF] curl.cpp:CheckIAMCredentialUpdate(1721): IAM Access Token refreshing... [INF] curl.cpp:GetIAMCredentials(2742): [IAM role=devops_test] **[CURL DBG] * Trying 169.254.169.254:80... <--- that IP can't be.** ``` #### /etc/fstab entry, if applicable ``` - 1st attempt using file not show mount point - 2nd attempt using role then can see but still blocked because command still hang-up using a wrong s3 public IP address s3fs on /mnt type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other) ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ```1st attempt using passwd-s3fs file: access denied 2nd attempt using assuming role (because use MFA) remains hung-up ``` ### Details about issue ``` it's strange using correct role, seems like is connecting but use a wrong IP address public for s3 and hung up without respond. and checking by shell using aws cli works using credentials with mfa obviusly aws s3 ls --region eu-south-1 --profile devops s3://upload-test 2021-01-23 13:42:32 0 s3-test.txt ```
Author
Owner

@juliogonzalez commented on GitHub (Jan 25, 2021):

@emper0r I am confused: are you using 1.87 or 1.88? The description contradicts itself. The Version section says 1.88, but the RPM package lists below says 1.87.

Right now EPEL7 only has 1.87, as 1.88 is still on the queue to be published (https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-8873b594c6)

<!-- gh-comment-id:766982176 --> @juliogonzalez commented on GitHub (Jan 25, 2021): @emper0r I am confused: are you using 1.87 or 1.88? The description contradicts itself. The `Version` section says 1.88, but the RPM package lists below says 1.87. Right now EPEL7 only has 1.87, as 1.88 is still on the queue to be published (https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-8873b594c6)
Author
Owner

@emper0r commented on GitHub (Jan 26, 2021):

Hi @juliogonzalez , I uninstalled 1.87 from rpm and I remain with source compiled 1.88 in any case is the same issue. hang-up trying connect to link-local IP address,

so uninstalling compiled version 1.88 and re-install rpm epel 1.87 same thing too.

s3fs keeps trying to connect to link-local
[INF] curl.cpp:RemakeHandle(2313): Retry request. [type=12][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/devop_test

Can you tell me with a clean system the correct steps to test it ? then I tell how was...

<!-- gh-comment-id:767402721 --> @emper0r commented on GitHub (Jan 26, 2021): Hi @juliogonzalez , I uninstalled 1.87 from rpm and I remain with source compiled 1.88 in any case is the same issue. hang-up trying connect to link-local IP address, so uninstalling compiled version 1.88 and re-install rpm epel 1.87 same thing too. s3fs keeps trying to connect to link-local [INF] curl.cpp:RemakeHandle(2313): Retry request. [type=12][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/devop_test Can you tell me with a clean system the correct steps to test it ? then I tell how was...
Author
Owner

@emper0r commented on GitHub (Jan 29, 2021):

Hi again, I reinstalled and create an user without mfa, directly access key and secret key, with another program I can access by api to my bucket, but with s3fs can't do.

the /etc/passwd-s3fs is correct and the error keep saying this

[CURL DBG] * Closing connection 1
[ERR] curl.cpp:RequestPerform(2282): HTTP response code 400, returning EIO. Body Text:
IllegalLocationConstraintExceptionThe eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.7553F1890C4B7B0A57LmdLLPdjpH/ZD3WIMxtx7p78hq/baq7/ogfWdyC8IEzWkrAPgrc3h27/c+cs1f4oK2Nu+N31Q=
[ERR] curl.cpp:CheckBucket(3344): Check bucket failed, S3 response:
IllegalLocationConstraintExceptionThe eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.7553F1890C4B7B0A57LmdLLPdjpH/ZD3WIMxtx7p78hq/baq7/ogfWdyC8IEzWkrAPgrc3h27/c+cs1f4oK2Nu+N31Q=
[CRT] s3fs.cpp:s3fs_check_service(3483): Bad Request(host=https://s3.amazonaws.com) - result of checking service.
[ERR] s3fs.cpp:s3fs_exit_fuseloop(3290): Exiting FUSE event loop due to errors

I test it in many ways... but can't connect with the bucket
any help ????????????????????????????????????

<!-- gh-comment-id:769727750 --> @emper0r commented on GitHub (Jan 29, 2021): Hi again, I reinstalled and create an user without mfa, directly access key and secret key, with another program I can access by api to my bucket, but with s3fs can't do. the /etc/passwd-s3fs is correct and the error keep saying this [CURL DBG] * Closing connection 1 [ERR] curl.cpp:RequestPerform(2282): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>IllegalLocationConstraintException</Code><Message>The eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.</Message><RequestId>7553F1890C4B7B0A</RequestId><HostId>57LmdLLPdjpH/ZD3WIMxtx7p78hq/baq7/ogfWdyC8IEzWkrAPgrc3h27/c+cs1f4oK2Nu+N31Q=</HostId></Error> [ERR] curl.cpp:CheckBucket(3344): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>IllegalLocationConstraintException</Code><Message>The eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent to.</Message><RequestId>7553F1890C4B7B0A</RequestId><HostId>57LmdLLPdjpH/ZD3WIMxtx7p78hq/baq7/ogfWdyC8IEzWkrAPgrc3h27/c+cs1f4oK2Nu+N31Q=</HostId></Error> [CRT] s3fs.cpp:s3fs_check_service(3483): Bad Request(host=https://s3.amazonaws.com) - result of checking service. [ERR] s3fs.cpp:s3fs_exit_fuseloop(3290): Exiting FUSE event loop due to errors I test it in many ways... but can't connect with the bucket any help ????????????????????????????????????
Author
Owner

@juliogonzalez commented on GitHub (Jan 30, 2021):

The eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent

The manual page for s3fs says:

endpoint (default="us-east-1")
sets the endpoint to use on signature version 4.
If this option is not specified, s3fs uses "us-east-1" region as the default.
If the s3fs could not connect to the region specified by this option, s3fs could not run.
But if you do not specify this option, and if you can not connect with the default region, s3fs will retry to automatically connect to the other region.
So s3fs can know the correct region name, because s3fs can find it in an error from the S3 server.

Try to add the end point for your bucket.

<!-- gh-comment-id:770243018 --> @juliogonzalez commented on GitHub (Jan 30, 2021): > The eu-south-1 location constraint is incompatible for the region specific endpoint this request was sent The manual page for s3fs says: ``` endpoint (default="us-east-1") sets the endpoint to use on signature version 4. If this option is not specified, s3fs uses "us-east-1" region as the default. If the s3fs could not connect to the region specified by this option, s3fs could not run. But if you do not specify this option, and if you can not connect with the default region, s3fs will retry to automatically connect to the other region. So s3fs can know the correct region name, because s3fs can find it in an error from the S3 server. ``` Try to add the end point for your bucket.
Author
Owner

@juliogonzalez commented on GitHub (Jan 30, 2021):

I don't know how the bucket was created, but check if this affects you:

https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html

If you send your create bucket request to the s3.amazonaws.com endpoint, the request goes to the us-east-1 Region. Accordingly, the signature calculations in Signature Version 4 must use us-east-1 as the Region, even if the location constraint in the request specifies another Region where the bucket is to be created. If you create a bucket in a Region other than US East (N. Virginia), your application must be able to handle 307 redirect. For more information, see Virtual hosting of buckets.

<!-- gh-comment-id:770243780 --> @juliogonzalez commented on GitHub (Jan 30, 2021): I don't know how the bucket was created, but check if this affects you: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html > If you send your create bucket request to the s3.amazonaws.com endpoint, the request goes to the us-east-1 Region. Accordingly, the signature calculations in Signature Version 4 must use us-east-1 as the Region, even if the location constraint in the request specifies another Region where the bucket is to be created. If you create a bucket in a Region other than US East (N. Virginia), your application must be able to handle 307 redirect. For more information, see Virtual hosting of buckets.
Author
Owner

@juliogonzalez commented on GitHub (Feb 7, 2021):

@emper0r did endpoint fix your problem?

<!-- gh-comment-id:774692900 --> @juliogonzalez commented on GitHub (Feb 7, 2021): @emper0r did `endpoint` fix your problem?
Author
Owner

@emper0r commented on GitHub (Feb 18, 2021):

no

<!-- gh-comment-id:781373343 --> @emper0r commented on GitHub (Feb 18, 2021): no
Author
Owner

@douglasjam commented on GitHub (May 5, 2023):

Same issue for me, only eu-south-1 fails

with sigv4

# s3fs ${S3FS_BUCKET_CH} ${USER_HOME}/ch -o allow_other -o umask=0002 -o uid=1001 -o gid=1001 -o nonempty -o url=https://s3.amazonaws.com -o endpoint=eu-south-1 -f
2023-05-05T11:47:25.822Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types
2023-05-05T11:47:25.825Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with GnuTLS(gcrypt)
2023-05-05T11:47:25.907Z [CRT] s3fs.cpp:s3fs_check_service(3502): Failed to connect by sigv4, so retry to connect by signature version 2.
2023-05-05T11:47:25.970Z [CRT] s3fs.cpp:s3fs_check_service(3514): Bad Request(host=https://s3.amazonaws.com) - result of checking service.

sigv2

# s3fs ${S3FS_BUCKET_CH} ${USER_HOME}/ch -o allow_other -o umask=0002 -o uid=1001 -o gid=1001 -o nonempty -o url=https://s3.amazonaws.com -o endpoint=eu-south-1 -o sigv2 -f
2023-05-05T11:47:39.801Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types
2023-05-05T11:47:39.803Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with GnuTLS(gcrypt)
2023-05-05T11:47:39.875Z [CRT] s3fs.cpp:s3fs_check_service(3514): Bad Request(host=https://s3.amazonaws.com) - result of checking service.
<!-- gh-comment-id:1536144348 --> @douglasjam commented on GitHub (May 5, 2023): Same issue for me, only eu-south-1 fails with sigv4 ``` # s3fs ${S3FS_BUCKET_CH} ${USER_HOME}/ch -o allow_other -o umask=0002 -o uid=1001 -o gid=1001 -o nonempty -o url=https://s3.amazonaws.com -o endpoint=eu-south-1 -f 2023-05-05T11:47:25.822Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types 2023-05-05T11:47:25.825Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with GnuTLS(gcrypt) 2023-05-05T11:47:25.907Z [CRT] s3fs.cpp:s3fs_check_service(3502): Failed to connect by sigv4, so retry to connect by signature version 2. 2023-05-05T11:47:25.970Z [CRT] s3fs.cpp:s3fs_check_service(3514): Bad Request(host=https://s3.amazonaws.com) - result of checking service. ``` sigv2 ``` # s3fs ${S3FS_BUCKET_CH} ${USER_HOME}/ch -o allow_other -o umask=0002 -o uid=1001 -o gid=1001 -o nonempty -o url=https://s3.amazonaws.com -o endpoint=eu-south-1 -o sigv2 -f 2023-05-05T11:47:39.801Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types 2023-05-05T11:47:39.803Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with GnuTLS(gcrypt) 2023-05-05T11:47:39.875Z [CRT] s3fs.cpp:s3fs_check_service(3514): Bad Request(host=https://s3.amazonaws.com) - result of checking service. ```
Author
Owner

@douglasjam commented on GitHub (May 5, 2023):

Found a way around

s3fs ${S3FS_BUCKET_CH} ${USER_HOME}/ch -o allow_other -o umask=0002 -o uid=1001 -o gid=1001 -o nonempty -o url=https://s3.eu-south-1.amazonaws.com -o endpoint=eu-south-1

<!-- gh-comment-id:1536156937 --> @douglasjam commented on GitHub (May 5, 2023): Found a way around ```s3fs ${S3FS_BUCKET_CH} ${USER_HOME}/ch -o allow_other -o umask=0002 -o uid=1001 -o gid=1001 -o nonempty -o url=https://s3.eu-south-1.amazonaws.com -o endpoint=eu-south-1```
Author
Owner

@ggtakec commented on GitHub (May 13, 2023):

Once #2155 is merged, if an error occurs at s3fs launching, it will become to output a message including suggestions about options(url/endpoint etc.).

@douglasjam
Thanks, I have reflected the results of your investigation.

<!-- gh-comment-id:1546704931 --> @ggtakec commented on GitHub (May 13, 2023): Once #2155 is merged, if an error occurs at s3fs launching, it will become to output a message including suggestions about options(url/endpoint etc.). @douglasjam Thanks, I have reflected the results of your investigation.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#805
No description provided.