[GH-ISSUE #1778] Use with IRSA

kerem commented

2026-03-04 01:49:53 +03:00

Owner

Originally created by @shaunc on GitHub (Oct 13, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1778

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

1.85

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.9

Kernel information (uname -r)

4.14.186-146.268.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Details about issue

I am looking for instructions for using with IRSA. In my pod container environment I have AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE. How can I get s3fs-fuse to consume these?

Originally created by @shaunc on GitHub (Oct 13, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1778 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) 1.85 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.9 #### Kernel information (uname -r) 4.14.186-146.268.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal ### Details about issue I am looking for instructions for using with IRSA. In my pod container environment I have AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE. How can I get s3fs-fuse to consume these?

kerem added the

help wanted

label

2026-03-04 01:49:53 +03:00

kerem commented

2026-03-04 01:49:54 +03:00

Author

Owner

@gaul commented on GitHub (Oct 14, 2021):

Sorry I'm not familiar with with IRSA. Later versions of s3fs do support IMDSv2 -- does this work for you?

@nmeyerhans do you know anything about this?

@gaul commented on GitHub (Oct 14, 2021): Sorry I'm not familiar with with IRSA. Later versions of s3fs do support IMDSv2 -- does this work for you? @nmeyerhans do you know anything about this?

kerem commented

2026-03-04 01:49:54 +03:00

Author

Owner

@nmeyerhans commented on GitHub (Oct 14, 2021):

Yes, IRSA is "IAM Roles for Service Accounts". It lets an application obtain temporary IAM role credentials based on a token obtained from some OIDC provider. You can read about it at https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ and the API you'd need to call is sts.AssumeRoleWithWebIdentity

The AWS C++ API supports this API. See its documentation for more details that might give some hints about how you might support it in s3fs.

@nmeyerhans commented on GitHub (Oct 14, 2021): Yes, IRSA is "IAM Roles for Service Accounts". It lets an application obtain temporary IAM role credentials based on a token obtained from some OIDC provider. You can read about it at https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/ and the API you'd need to call is [sts.AssumeRoleWithWebIdentity](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html) The AWS C++ API supports this API. See [its documentation](https://sdk.amazonaws.com/cpp/api/LATEST/class_aws_1_1_s_t_s_1_1_s_t_s_client.html#a6e66e152a726649a4f15c86a875b0e04) for more details that might give some hints about how you might support it in s3fs.

kerem commented

2026-03-04 01:49:54 +03:00

Author

Owner

@theis188 commented on GitHub (Dec 7, 2021):

Thanks for opening this issue, I would also find this feature useful.

@theis188 commented on GitHub (Dec 7, 2021): Thanks for opening this issue, I would also find this feature useful.

kerem commented

2026-03-04 01:49:54 +03:00

Author

Owner

@albpal commented on GitHub (Jun 14, 2022):

+1. This feature will help us also.

@albpal commented on GitHub (Jun 14, 2022): +1. This feature will help us also.

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@aru-trackunit commented on GitHub (Sep 1, 2022):

+1

@aru-trackunit commented on GitHub (Sep 1, 2022): +1

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@bressanmarcos commented on GitHub (Oct 29, 2022):

👍 +1

@bressanmarcos commented on GitHub (Oct 29, 2022): :+1: +1

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@mak-454 commented on GitHub (Nov 4, 2022):

+1

@mak-454 commented on GitHub (Nov 4, 2022): +1

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@yao531441 commented on GitHub (Jan 17, 2023):

+1

@yao531441 commented on GitHub (Jan 17, 2023): +1

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@yukinakanaka commented on GitHub (Jan 18, 2023):

+1

@yukinakanaka commented on GitHub (Jan 18, 2023): +1

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 22, 2023):

As one solution to this issue:

I have released the following source as a prototype few month ago.
https://github.com/ggtakec/s3fs-fuse-awscred-lib
You can refer to the README.md for how to build/use it.
This library is a DSO that wraps the aws-sdk-cpp authentication functionality.
The code in the master branch of s3fs-fuse includes an option(credlib) to delegate authentication to an external DSO library.
By specifying this library, I think that functions such as authentication tokens provided by AWS works.

If anyone has tried it, I'd love to know the results.
Could anyone use the code from the master branch of s3fs-fuse to try out the libs3fsawscred.so file that the above repository creates?

@ggtakec commented on GitHub (Jan 22, 2023): As one solution to this issue: I have released the following source as a prototype few month ago. https://github.com/ggtakec/s3fs-fuse-awscred-lib _You can refer to the README.md for how to build/use it._ This library is a DSO that wraps the `aws-sdk-cpp` authentication functionality. The code in the master branch of s3fs-fuse includes an option(`credlib`) to delegate authentication to an external DSO library. By specifying this library, I think that functions such as authentication tokens provided by AWS works. If anyone has tried it, I'd love to know the results. Could anyone use the code from the master branch of s3fs-fuse to try out the `libs3fsawscred.so` file that the above repository creates?

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@aidandj commented on GitHub (Jan 27, 2023):

I tried it and ended up needing to make these changes for it to work: https://github.com/artificialinc/s3fs-fuse/tree/aidan/ext-creds

It was failing early for me, and then the session token wasn't being set

@aidandj commented on GitHub (Jan 27, 2023): I tried it and ended up needing to make these changes for it to work: https://github.com/artificialinc/s3fs-fuse/tree/aidan/ext-creds It was failing early for me, and then the session token wasn't being set

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 29, 2023):

@artificial-aidan Thanks for your kindness.
As you pointed out, it was a buggy code.
I am creating a PR #2101 based on your code.(If merged, it will be reflected in the master code.)

If possible, please let me know about It was failing early that you tried in your environment.
I would like to know the log when the dbglevel=info(and curldbg: but this option makes log file large) option is given when starting s3fs.
Also, although the log becomes complicated, if you add credlib_opts=Info along with credlib=libs3fsawscred.so, you can also record the trace log of libs3fsawscred.so.

@ggtakec commented on GitHub (Jan 29, 2023): @artificial-aidan Thanks for your kindness. As you pointed out, it was a buggy code. I am creating a PR #2101 based on your code.(If merged, it will be reflected in the master code.) If possible, please let me know about `It was failing early` that you tried in your environment. I would like to know the log when the `dbglevel=info`(and `curldbg`: but this option makes log file large) option is given when starting s3fs. Also, although the log becomes complicated, if you add `credlib_opts=Info` along with `credlib=libs3fsawscred.so`, you can also record the trace log of `libs3fsawscred.so`.

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@aidandj commented on GitHub (Jan 29, 2023):

Yeah I should be able to reproduce it. I will look back at my logs on Monday.

@aidandj commented on GitHub (Jan 29, 2023): Yeah I should be able to reproduce it. I will look back at my logs on Monday.

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@aidandj commented on GitHub (Jan 30, 2023):

Reverting back to the old code was more work than I had time for.

I ran into 2 issues. The first was the error could not determine how to establish security credentials. This is because when using IRSA, like ECS, we need to delay the loading of the credentials. This might cause issues if you were using the external library to load other credentials, I'm not sure. But for IRSA, it needs to be delayed

The second issue was the error message was when I actually got the creds, but then s3fs tried to make a call, and I got a permission denied error. This error turned out to be from not setting the session token.

Here is the output of a successful run:

FUSE library version: 2.9.9
nullpath_ok: 0
nopath: 0
utime_omit_ok: 1
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.31
flags=0x03fffffb
max_readahead=0x00020000
[s3fsawscred] : Access Key Id = XXXXXXXX
[s3fsawscred] : Secret Key    = XXXXXXXX
[s3fsawscred] : Session Token = XXXXXXXXX
[s3fsawscred] : expiration    = 2023-01-30TXXXXXX
   INIT: 7.19
   flags=0x00000011
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   unique: 2, success, outsize: 40

@aidandj commented on GitHub (Jan 30, 2023): Reverting back to the old code was more work than I had time for. I ran into 2 issues. The first was the error `could not determine how to establish security credentials.` This is because when using IRSA, like ECS, we need to delay the loading of the credentials. This might cause issues if you were using the external library to load other credentials, I'm not sure. But for IRSA, it needs to be delayed The second issue was the error message was when I actually got the creds, but then s3fs tried to make a call, and I got a permission denied error. This error turned out to be from not setting the session token. Here is the output of a successful run: ``` FUSE library version: 2.9.9 nullpath_ok: 0 nopath: 0 utime_omit_ok: 1 unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0 INIT: 7.31 flags=0x03fffffb max_readahead=0x00020000 [s3fsawscred] : Access Key Id = XXXXXXXX [s3fsawscred] : Secret Key = XXXXXXXX [s3fsawscred] : Session Token = XXXXXXXXX [s3fsawscred] : expiration = 2023-01-30TXXXXXX INIT: 7.19 flags=0x00000011 max_readahead=0x00020000 max_write=0x00020000 max_background=0 congestion_threshold=0 unique: 2, success, outsize: 40 ```

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 5, 2023):

@artificial-aidan Sorry about the late reply.

Reverting back to the old code was more work than I had time for.

Thank you for your great help.

Since there was a correction omission earlier, I corrected #2101 a little.

And I would like to reconfirm with you whether this problem still remains or not.
Did #2101(or the modified code you originally tried) solve this problems you had?
Or are you still getting session token not set, etc?

I would like to merge #2101 if it solves this problem.
If this problem still remains, I will prepare the environment to use IRSA and investigate, so it will take some time.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Feb 5, 2023): @artificial-aidan Sorry about the late reply. > Reverting back to the old code was more work than I had time for. Thank you for your great help. Since there was a correction omission earlier, I corrected #2101 a little. And I would like to reconfirm with you whether this problem still remains or not. Did #2101(or the modified code you originally tried) solve this problems you had? Or are you still getting `session token not set`, etc? I would like to merge #2101 if it solves this problem. If this problem still remains, I will prepare the environment to use IRSA and investigate, so it will take some time. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:49:55 +03:00

Author

Owner

@aidandj commented on GitHub (Feb 5, 2023):

Let me try it out Monday.

The branch I shared was fully functional for me.

@aidandj commented on GitHub (Feb 5, 2023): Let me try it out Monday. The branch I shared was fully functional for me.

kerem commented

2026-03-04 01:49:56 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 18, 2023):

@artificial-aidan
#2101 has been merged.
Please check and let me know if you have any problems.

@ggtakec commented on GitHub (Feb 18, 2023): @artificial-aidan #2101 has been merged. Please check and let me know if you have any problems.

kerem commented

2026-03-04 01:49:56 +03:00

Author

Owner

@bmihaescu commented on GitHub (May 22, 2023):

@ggtakec I also have enabled IRSA on an EKS pod and it seems that it's still trying to get the role/credentials from instance metadata. What do I need to change to the s3fs command to use the changes you merged with #2101?

I use V1.92 version of s3fs

Thanks

@bmihaescu commented on GitHub (May 22, 2023): @ggtakec I also have enabled IRSA on an EKS pod and it seems that it's still trying to get the role/credentials from instance metadata. What do I need to change to the s3fs command to use the changes you merged with #2101? I use V1.92 version of s3fs Thanks

kerem commented

2026-03-04 01:49:56 +03:00

Author

Owner

@ggtakec commented on GitHub (Jun 4, 2023):

@bmihaescu
Currently, s3fs does not have IRSA authentication functionality.
Options(credlib and credlib_ops) added in #1927(#2101) are the ability to load external authentication libraries into s3fs.
Currently, it seems that the only library that supports this is s3fs-fuse-awscred-lib.
For how to use this s3fs-fuse-awscred-lib library refer to its README.
This library uses the aws-sdk-app and supports all authentication methods provided by AWS(except SSO).
If you have any questions about this library, please open as an issue on s3fs-fuse-awscred-lib github.
Please let me know if I misunderstood your question.

@ggtakec commented on GitHub (Jun 4, 2023): @bmihaescu Currently, s3fs does not have IRSA authentication functionality. Options(`credlib` and `credlib_ops`) added in #1927(#2101) are the ability to load external authentication libraries into s3fs. Currently, it seems that the only library that supports this is [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib). For how to use this [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib) library refer to its README. This library uses the `aws-sdk-app` and supports all authentication methods provided by AWS(except SSO). If you have any questions about this library, please open as an issue on [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib) github. Please let me know if I misunderstood your question.

kerem referenced this issue

2026-03-04 02:01:45 +03:00

[PR #923] [MERGED] Reverted automatic region change and changed messages #1702

Rows
Columns

[GH-ISSUE #1778] Use with IRSA #916

Additional Information

Version of s3fs being used (s3fs --version)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Kernel information (uname -r)

GNU/Linux Distribution, if applicable (cat /etc/os-release)

Details about issue