starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1601] s3fs not using EKS pod's service-account/IAM Role for accessing S3 #839

New issue
Closed
opened 2026-03-04 01:49:14 +03:00 by kerem · 11 comments
kerem commented 2026-03-04 01:49:14 +03:00
Owner
Copy link

Originally created by @kadambkaluskar on GitHub (Mar 11, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1601

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.80(commit:unknown) with GnuTLS(gcrypt)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Version: 2.9.7-1+deb9u2

Kernel information (uname -r)

4.14.219-161.340.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

s3fs command line used, if applicable

s3fs -f -o use_path_request_style -o iam_role="<iam-role-name>" -o url="https://s3-us-west-1.amazonaws.com" -o endpoint=us-west-1 -o dbglevel=debug -o curldbg -o allow_other   <bucket-name>:<prefix> <mount-path>

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages


[CRT] s3fs.cpp:set_s3fs_log_level(253): change debug level from [CRT] to [DBG] 
[INF]     s3fs.cpp:set_moutpoint_attribute(4151): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[CRT] s3fs.cpp:s3fs_init(3334): init v1.80(commit:unknown) with GnuTLS(gcrypt)
[INF] s3fs.cpp:s3fs_check_service(3708): check services.
[INF]       curl.cpp:GetIAMCredentials(2306): [IAM role=<iam-role>]
[DBG] curl.cpp:GetHandler(272): Get handler from pool: 31
[DBG] curl.cpp:RequestPerform(1893): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/<iam-role>
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/iam/security-credentials/<iam-role> HTTP/1.1
Host: 169.254.169.254
User-Agent: s3fs/1.80 (commit hash unknown; GnuTLS(gcrypt))
Accept: */*

< HTTP/1.1 404 Not Found
< Content-Type: text/html
< Content-Length: 339
< Date: Thu, 11 Mar 2021 08:39:56 GMT
< Server: EC2ws
< Connection: close
< 
* Curl_http_done: called premature == 0
* Closing connection 0
[INF]       curl.cpp:RequestPerform(1932): HTTP response code 404 was returned, returning ENOENT
[DBG] curl.cpp:RequestPerform(1933): Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
		 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>

[DBG] curl.cpp:ReturnHandler(295): Return handler to pool: 31
[CRT] s3fs.cpp:s3fs_check_service(3712): Failed to check IAM role name(<iam-role>).
[ERR] s3fs.cpp:s3fs_exit_fuseloop(3324): Exiting FUSE event loop due to errors

[INF] s3fs.cpp:s3fs_destroy(3395): destroy
[WAN] s3fs.cpp:s3fs_destroy(3399): Could not release curl library.

Details about the issue

I am trying to mount s3 bucket inside my Kubernetes pod which is running on EKS.
To give S3 access to Pod, I have created an IAM role, and attached it to a service account, and using the same for s3fs.
But it seems the s3fs utility calls EC2 metadata URL, where it doesn't find the mentioned IAM, but the IAM role for EKS Node.

How can I mount an S3 bucket inside my EKS pod, with the help of the IAM role/Service Account?
This works when I use the access-keys for the same user, but I don't want to use the access-keys due to security reasons.

Originally created by @kadambkaluskar on GitHub (Mar 11, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1601 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.80(commit:unknown) with GnuTLS(gcrypt) #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Version: 2.9.7-1+deb9u2 #### Kernel information (uname -r) 4.14.219-161.340.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) PRETTY_NAME="Debian GNU/Linux 9 (stretch)" NAME="Debian GNU/Linux" VERSION_ID="9" VERSION="9 (stretch)" VERSION_CODENAME=stretch ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" #### s3fs command line used, if applicable ``` s3fs -f -o use_path_request_style -o iam_role="<iam-role-name>" -o url="https://s3-us-west-1.amazonaws.com" -o endpoint=us-west-1 -o dbglevel=debug -o curldbg -o allow_other <bucket-name>:<prefix> <mount-path> ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` [CRT] s3fs.cpp:set_s3fs_log_level(253): change debug level from [CRT] to [DBG] [INF] s3fs.cpp:set_moutpoint_attribute(4151): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [CRT] s3fs.cpp:s3fs_init(3334): init v1.80(commit:unknown) with GnuTLS(gcrypt) [INF] s3fs.cpp:s3fs_check_service(3708): check services. [INF] curl.cpp:GetIAMCredentials(2306): [IAM role=<iam-role>] [DBG] curl.cpp:GetHandler(272): Get handler from pool: 31 [DBG] curl.cpp:RequestPerform(1893): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/<iam-role> * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) > GET /latest/meta-data/iam/security-credentials/<iam-role> HTTP/1.1 Host: 169.254.169.254 User-Agent: s3fs/1.80 (commit hash unknown; GnuTLS(gcrypt)) Accept: */* < HTTP/1.1 404 Not Found < Content-Type: text/html < Content-Length: 339 < Date: Thu, 11 Mar 2021 08:39:56 GMT < Server: EC2ws < Connection: close < * Curl_http_done: called premature == 0 * Closing connection 0 [INF] curl.cpp:RequestPerform(1932): HTTP response code 404 was returned, returning ENOENT [DBG] curl.cpp:RequestPerform(1933): Body Text: <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>404 - Not Found</title> </head> <body> <h1>404 - Not Found</h1> </body> </html> [DBG] curl.cpp:ReturnHandler(295): Return handler to pool: 31 [CRT] s3fs.cpp:s3fs_check_service(3712): Failed to check IAM role name(<iam-role>). [ERR] s3fs.cpp:s3fs_exit_fuseloop(3324): Exiting FUSE event loop due to errors [INF] s3fs.cpp:s3fs_destroy(3395): destroy [WAN] s3fs.cpp:s3fs_destroy(3399): Could not release curl library. ``` ### Details about the issue I am trying to mount s3 bucket inside my Kubernetes pod which is running on EKS. To give S3 access to Pod, I have created an IAM role, and attached it to a service account, and using the same for s3fs. But it seems the s3fs utility calls EC2 metadata URL, where it doesn't find the mentioned IAM, but the IAM role for EKS Node. How can I mount an S3 bucket inside my EKS pod, with the help of the IAM role/Service Account? This works when I use the access-keys for the same user, but I don't want to use the access-keys due to security reasons.
kerem 2026-03-04 01:49:14 +03:00
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@kobi-toptal commented on GitHub (Oct 26, 2021):

i am having the same issue, any updates on this?

<!-- gh-comment-id:952114396 --> @kobi-toptal commented on GitHub (Oct 26, 2021): i am having the same issue, any updates on this?
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@elia-netroadshow commented on GitHub (Nov 1, 2021):

Seems to be happening here as well

<!-- gh-comment-id:956746945 --> @elia-netroadshow commented on GitHub (Nov 1, 2021): Seems to be happening here as well
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@elia-netroadshow commented on GitHub (Nov 1, 2021):

Also in our case, the service makes the whole container instance non-responsive

<!-- gh-comment-id:956748369 --> @elia-netroadshow commented on GitHub (Nov 1, 2021): Also in our case, the service makes the whole container instance non-responsive
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@shidaxi commented on GitHub (Dec 26, 2021):

Met the same issue, any updates on this?

<!-- gh-comment-id:1001253714 --> @shidaxi commented on GitHub (Dec 26, 2021): Met the same issue, any updates on this?
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@Confushion commented on GitHub (Dec 29, 2021):

Same issue here when trying to use EKS OIDC Provider and ServiceAccount annotation.

It does however work with Kube2iam and pod annotation, but that's just what I try to get rid of...

<!-- gh-comment-id:1002352001 --> @Confushion commented on GitHub (Dec 29, 2021): Same issue here when trying to use EKS OIDC Provider and ServiceAccount annotation. It does however work with Kube2iam and pod annotation, but that's just what I try to get rid of...
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@kobi-toptal commented on GitHub (Dec 29, 2021):

I have migrated to goofys, the code from master branch has updated AWS sdk. Aws sdk automatically handles the authentication.

<!-- gh-comment-id:1002563766 --> @kobi-toptal commented on GitHub (Dec 29, 2021): I have migrated to goofys, the code from master branch has updated AWS sdk. Aws sdk automatically handles the authentication.
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@rma945 commented on GitHub (Jan 20, 2022):

Looks like the whole auth mechanics in s3fs were done manually, so s3fs doesn't know anything about IRSA-based auth and don't use
the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE parameters. So this issue can be fixed if the s3fs will migrate to the AWS C++ SDK as described in this issue #1068

<!-- gh-comment-id:1017141931 --> @rma945 commented on GitHub (Jan 20, 2022): Looks like the whole auth mechanics in s3fs were done manually, so s3fs doesn't know anything about IRSA-based auth and don't use the **AWS_ROLE_ARN** and **AWS_WEB_IDENTITY_TOKEN_FILE** parameters. So this issue can be fixed if the **s3fs** will migrate to the AWS C++ SDK as described in this issue #1068
kerem commented 2026-03-04 01:49:15 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jan 23, 2022):

Moving to the C++ SDK is a heavy lift and unlikely to be done soon.

@nmeyerhans can you comment on IRSA? Does this relate to IMDSv2?

<!-- gh-comment-id:1019595227 --> @gaul commented on GitHub (Jan 23, 2022): Moving to the C++ SDK is a heavy lift and unlikely to be done soon. @nmeyerhans can you comment on IRSA? Does this relate to IMDSv2?
kerem commented 2026-03-04 01:49:16 +03:00
Author
Owner
Copy link

@nmeyerhans commented on GitHub (Jan 24, 2022):

No, this is completely unrelated to IMDSv2, which is basically a small modification to how a program interacts with the existing IMDS APIs. IRSA is a mechanism to allow IAM roles to be associated with Kubernetes pods. From the s3fs applications point of view, the API you need to use is sts.AssumeRoleWithWebIdentity. See also this blog post describing how it works.

Also, support for this has previously been requested in #1778.

<!-- gh-comment-id:1019719677 --> @nmeyerhans commented on GitHub (Jan 24, 2022): No, this is completely unrelated to IMDSv2, which is basically a small modification to how a program interacts with the existing IMDS APIs. IRSA is a mechanism to allow IAM roles to be associated with Kubernetes pods. From the s3fs applications point of view, the API you need to use is [sts.AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html). See also [this blog post](https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/) describing how it works. Also, support for this has previously been requested in #1778.
kerem commented 2026-03-04 01:49:16 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 26, 2022):

It is not perfect for incorporating AWS-SDK-CPP, but I think it can Incorporate AWS-SDK-CPP authentication(loading AccessKey and SessionToken, etc) for the current s3fs-fuse.
For more information, I would like to discuss this with @gaul.
So please wait for a while.

<!-- gh-comment-id:1022199522 --> @ggtakec commented on GitHub (Jan 26, 2022): It is not perfect for incorporating `AWS-SDK-CPP`, but I think it can Incorporate `AWS-SDK-CPP` authentication(loading AccessKey and SessionToken, etc) for the current s3fs-fuse. For more information, I would like to discuss this with @gaul. So please wait for a while.
kerem commented 2026-03-04 01:49:16 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jan 27, 2022):

Duplicate of #1778. Thank you for deduplicating this, @nmeyerhans!

<!-- gh-comment-id:1023215128 --> @gaul commented on GitHub (Jan 27, 2022): Duplicate of #1778. Thank you for deduplicating this, @nmeyerhans!
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#839
No description provided.