[GH-ISSUE #1290 ] Credential refresh from without restarting s3fs process #692

Author

Owner

@gaul commented on GitHub (May 18, 2020):

@michallula could you look into this?

@gaul commented on GitHub (May 18, 2020): @michallula could you look into this?

kerem commented

Author

Owner

@anvishwakarma commented on GitHub (Jun 3, 2020):

Is there any update on this issue?

@anvishwakarma commented on GitHub (Jun 3, 2020): Is there any update on this issue?

kerem commented

Author

Owner

@gilbeckers commented on GitHub (Apr 29, 2021):

Any update on this issue? We would also be interested in refreshing the credentials without restarting the s3fs process.

@gilbeckers commented on GitHub (Apr 29, 2021): Any update on this issue? We would also be interested in refreshing the credentials without restarting the s3fs process.

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 25, 2021):

Would re-reading the credentials file periodically be enough to address this issue?

@gaul commented on GitHub (Jul 25, 2021): Would re-reading the credentials file periodically be enough to address this issue?

kerem commented

Author

Owner

@maresb commented on GitHub (Jul 25, 2021):

Would re-reading the credentials file periodically be enough to address this issue?

@gaul, in my case, yes.

@maresb commented on GitHub (Jul 25, 2021): > Would re-reading the credentials file periodically be enough to address this issue? @gaul, in my case, yes.

kerem commented

https://docs.aws.amazon.com/sdkref/latest/guide/setting-global-duration_seconds.html

Author

Owner

@gaul commented on GitHub (Jul 25, 2021):

How often does the AWS SDK refresh its credentials? Does duration_seconds describe this or is it something else:

@gaul commented on GitHub (Jul 25, 2021): How often does the AWS SDK refresh its credentials? Does `duration_seconds` describe this or is it something else: https://docs.aws.amazon.com/sdkref/latest/guide/setting-global-duration_seconds.html

kerem commented

Author

Owner

@maresb commented on GitHub (Jul 25, 2021):

@gaul, if only there were a simple answer...

In my case, I'd need 300 seconds:

AWS managed temporary credentials are updated under any of the following conditions:

Whenever a certain period of time passes. Currently, this is every five minutes.

...

And duration_seconds is not defined in the ~/.aws/credentials file which I am provided 🤦‍♂️

@maresb commented on GitHub (Jul 25, 2021): @gaul, if only there were a simple answer... In [my case](https://docs.aws.amazon.com/cloud9/latest/user-guide/how-cloud9-with-iam.html#auth-and-access-control-temporary-managed-credentials), I'd need 300 seconds: > AWS managed temporary credentials are updated under any of the following conditions: > * Whenever a certain period of time passes. Currently, this is every five minutes. > * ... And `duration_seconds` is not defined in the `~/.aws/credentials` file which I am provided :man_facepalming:

kerem commented

Author

Owner

@maresb commented on GitHub (Jul 25, 2021):

And AWS can (and does) change this value when they feel like it.

@maresb commented on GitHub (Jul 25, 2021): And AWS can (and does) change this value when they feel like it.

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 25, 2021):

@nmeyerhans Any chance you could look at this? This appears to be one of the most common AWS issues.

@gaul commented on GitHub (Jul 25, 2021): @nmeyerhans Any chance you could look at this? This appears to be one of the most common AWS issues.

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 27, 2021):

Related to #1122.

@gaul commented on GitHub (Jul 27, 2021): Related to #1122.

kerem commented

Author

Owner

@nmeyerhans commented on GitHub (Sep 2, 2021):

~/.aws/credentials is used to store static credentials. There's no provision for providing expiration times for the creds in that file, as they don't typically expire, and the contents of the file are typically not re-read. That's the case with s3fs and with the AWS SDKs.

If you want to use credentials that expire, you should be using the IMDS to retrieve them. The credentials provided by IMDS include an expiration data and can be refreshed automatically by s3fs.

@nmeyerhans commented on GitHub (Sep 2, 2021): `~/.aws/credentials` is used to store static credentials. There's no provision for providing expiration times for the creds in that file, as they don't typically expire, and the contents of the file are typically not re-read. That's the case with s3fs and with the AWS SDKs. If you want to use credentials that expire, you should be using the IMDS to retrieve them. The credentials provided by IMDS include an expiration data and can be refreshed automatically by s3fs.

kerem commented

Author

Owner

@maresb commented on GitHub (Sep 3, 2021):

@nmeyerhans, I don't disagree, but in an unfortunate case of AWS violating their own best-practices, the temporary credentials for the Cloud9 AWS service are delivered via ~/.aws/credentials. (For more details see the link I posted above.)

One could easily argue that this is a bug in Cloud9 rather than s3fs. Indeed I recently migrated away from Cloud9, primarily due to the problems caused by the Cloud9 credential refresh process (not only in connection with s3fs). Thus this issue no longer affects me.

@maresb commented on GitHub (Sep 3, 2021): @nmeyerhans, I don't disagree, but in an unfortunate case of AWS violating their own best-practices, the temporary credentials for the Cloud9 AWS service are delivered via `~/.aws/credentials`. (For more details see the link I posted above.) One could easily argue that this is a bug in Cloud9 rather than s3fs. Indeed I recently migrated away from Cloud9, primarily due to the problems caused by the Cloud9 credential refresh process (not only in connection with s3fs). Thus this issue no longer affects me.

kerem commented

Author

Owner

@Nintorac commented on GitHub (May 31, 2022):

I am using https://pypi.org/project/aws-mfa/

This inserts temporary creds into the credentials file upon authentication and then after they expire s3fs fails to work. I would consider this a fairly typical use case since it's the cleanest, least effort way I've found to deal with MFA in AWS...Though happy to be pointed a better alternative if one exists.

Otherwise is there a way to manually trigger a refresh without restarting the process perhaps?

@Nintorac commented on GitHub (May 31, 2022): I am using https://pypi.org/project/aws-mfa/ This inserts temporary creds into the credentials file upon authentication and then after they expire s3fs fails to work. I would consider this a fairly typical use case since it's the cleanest, least effort way I've found to deal with MFA in AWS...Though happy to be pointed a better alternative if one exists. Otherwise is there a way to manually trigger a refresh without restarting the process perhaps?

kerem commented

Author

Owner

@ggtakec commented on GitHub (Jun 2, 2022):

I am currently submitting #1927 as a solution to this problem.
That PR provides new option(credlib) to delegate the credential processing used by s3fs to an external DSO library.
I also prepared sample DSO library that wraps the authentication process by aws-sdk-cpp as an external DSO for AWS.

If this option is specified(to use an external authentication DSO), it will check the token's expiration date, and if it has timed out, it will authentication process by the DSO again and refresh the token.

You can get that s3fs code from https://github.com/ggtakec/s3fs-fuse/tree/extcred_mod for the s3fs code(in my branch because it has not been merged).
And sample DSO, get from https://github.com/ggtakec/s3fs-fuse-awscred-lib .
I think you can try these right away if you can build them.

The code is still under review, but I'd be happy if anyone could give it a try.

@ggtakec commented on GitHub (Jun 2, 2022): I am currently submitting #1927 as a solution to this problem. That PR provides new option(`credlib`) to delegate the credential processing used by s3fs to an external DSO library. I also prepared sample DSO library that wraps the authentication process by `aws-sdk-cpp` as an external DSO for AWS. If this option is specified(to use an external authentication DSO), it will check the token's expiration date, and if it has timed out, it will authentication process by the DSO again and refresh the token. You can get that s3fs code from https://github.com/ggtakec/s3fs-fuse/tree/extcred_mod for the s3fs code(in my branch because it has not been merged). And sample DSO, get from https://github.com/ggtakec/s3fs-fuse-awscred-lib . I think you can try these right away if you can build them. The code is still under review, but I'd be happy if anyone could give it a try.

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 22, 2022):

Sorry for the delay in replying.

I merged PR #1927 into master branch.
I hope this fix will be one of the solutions for this issue.

Details are provided in the comments of Issue #1068 below:
https://github.com/s3fs-fuse/s3fs-fuse/issues/1068#issuecomment-1287790082

Could anyone please try this?

@ggtakec commented on GitHub (Oct 22, 2022): _Sorry for the delay in replying._ I merged PR #1927 into master branch. I hope this fix will be one of the solutions for this issue. Details are provided in the comments of Issue #1068 below: https://github.com/s3fs-fuse/s3fs-fuse/issues/1068#issuecomment-1287790082 Could anyone please try this?

kerem commented

Author

Owner

@sheromon commented on GitHub (Jun 23, 2023):

@ggtakec, thank you for this contribution! I have been facing this same problem where I have temporary credentials, and they expire, and I lose access to my mounted folder and have to run s3fs again to have it start working.

I found your project https://github.com/ggtakec/s3fs-fuse-awscred-lib and installed it in my Docker image, and I have tried it out. I found that I am able to use it as described in the README to mount my bucket to a directory. However, I found that when my credentials expire, I still lose access and have to run s3fs again, just like how it worked with a password file. This suggests to me that s3fs-fuse does not automatically refresh credentials when the current credentials expire. Is that the expected behavior?

@sheromon commented on GitHub (Jun 23, 2023): @ggtakec, thank you for this contribution! I have been facing this same problem where I have temporary credentials, and they expire, and I lose access to my mounted folder and have to run s3fs again to have it start working. I found your project https://github.com/ggtakec/s3fs-fuse-awscred-lib and installed it in my Docker image, and I have tried it out. I found that I am able to use it as described in the README to mount my bucket to a directory. However, I found that when my credentials expire, I still lose access and have to run s3fs again, just like how it worked with a password file. This suggests to me that s3fs-fuse does not automatically refresh credentials when the current credentials expire. Is that the expected behavior?

kerem commented

Author

Owner

@ggtakec commented on GitHub (Jun 25, 2023):

@sheromon s3fs-fuse-awscred-lib excludes support for temporary credentials such as SSO.
Is it that SSO or other credential that you are expiring?
A discussion of this is in #2165.

@ggtakec commented on GitHub (Jun 25, 2023): @sheromon [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib) excludes support for temporary credentials such as SSO. Is it that SSO or other credential that you are expiring? A discussion of this is in #2165.

kerem commented

Author

Owner

@sheromon commented on GitHub (Jun 26, 2023):

@ggtakec, ah, sorry, I am not familiar with "AWS SSO". I looked it up, and I see that they renamed it to the IAM identity center, but I'm still not clear on what that includes. However, I can explain what AWS credentials methods I was trying to use.

One method I tried was setting AWS_CONTAINER_CREDENTIALS_FULL_URI to my STS endpoint. It looked like s3fs-fuse-awscred-lib was doing the right thing, but ultimately, this didn't work for me because instead of using AWS STS AssumeRoleWithWebIdentity, I was using MinIO STS AssumeRoleWithWebIdentity. I feel like MinIO STS AssumeRoleWithWebIdentity should work the same as the AWS one, but it looks to me that it is different enough that the code in the AWS C++ SDK was not compatible.

Because my first method didn't work, and I still wanted to experiment with s3fs-fuse-awscred-lib, I tried writing my AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN into the file at ~/.aws/credentials, running the s3fs command to mount my bucket to a directory, and then 5 minutes before the token will expire, rewriting the ~/.aws/credentials file with new credentials. This is when I observed that the initial mount worked, but after the first token expired, I could no longer ls my mounted directory until I reran s3fs to re-mount the bucket. The way I was getting the credentials to put in the file was using MinIO STS AssumeRoleWithWebIdentity, which I called myself to get the values to write in the file.

@sheromon commented on GitHub (Jun 26, 2023): @ggtakec, ah, sorry, I am not familiar with "AWS SSO". I looked it up, and I see that they renamed it to the IAM identity center, but I'm still not clear on what that includes. However, I can explain what AWS credentials methods I was trying to use. One method I tried was setting `AWS_CONTAINER_CREDENTIALS_FULL_URI` to my STS endpoint. It looked like [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib) was doing the right thing, but ultimately, this didn't work for me because instead of using AWS STS AssumeRoleWithWebIdentity, I was using MinIO STS AssumeRoleWithWebIdentity. I feel like MinIO STS AssumeRoleWithWebIdentity should work the same as the AWS one, but it looks to me that it is different enough that the code in the AWS C++ SDK was not compatible. Because my first method didn't work, and I still wanted to experiment with [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib), I tried writing my `AWS_ACCESS_KEY_ID`, ` AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` into the file at `~/.aws/credentials`, running the s3fs command to mount my bucket to a directory, and then 5 minutes before the token will expire, rewriting the `~/.aws/credentials` file with new credentials. This is when I observed that the initial mount worked, but after the first token expired, I could no longer `ls` my mounted directory until I reran s3fs to re-mount the bucket. The way I was getting the credentials to put in the file was using MinIO STS AssumeRoleWithWebIdentity, which I called myself to get the values to write in the file.

kerem commented

Author

Owner

@sheromon commented on GitHub (Jun 27, 2023):

@ggtakec, I think I may have found why I am having a problem with my credentials not being automatically refreshed. In my logging output, I saw the following (with the credentials replaced by Xs).

[s3fsawscred] : Access Key Id = XXXXX
[s3fsawscred] : Secret Key    = XXXXX
[s3fsawscred] : Session Token = XXXXX
[s3fsawscred] : expiration    = 2262-04-11T23:47:16Z

I set my token to expire in around 15 minutes, so the printed expiration date/time is not correct. I am thinking that s3fs-fuse is getting the incorrect expiration date, so it does not think it needs to refresh credentials after 15 minutes have passed, and instead it just tries to use the same credentials, and they stop working after 15 minutes.

I forked the repo for s3fs-fuse-awscred-lib and added a line of logging, so I can see that my expiration milliseconds is the wrong value here: https://github.com/ggtakec/s3fs-fuse-awscred-lib/blob/master/awscred_func.cpp#L162. I am not sure why. If I decode my token, it has the right expiration timestamp.

@sheromon commented on GitHub (Jun 27, 2023): @ggtakec, I think I may have found why I am having a problem with my credentials not being automatically refreshed. In my logging output, I saw the following (with the credentials replaced by Xs). ``` [s3fsawscred] : Access Key Id = XXXXX [s3fsawscred] : Secret Key = XXXXX [s3fsawscred] : Session Token = XXXXX [s3fsawscred] : expiration = 2262-04-11T23:47:16Z ``` I set my token to expire in around 15 minutes, so the printed expiration date/time is not correct. I am thinking that s3fs-fuse is getting the incorrect expiration date, so it does not think it needs to refresh credentials after 15 minutes have passed, and instead it just tries to use the same credentials, and they stop working after 15 minutes. I forked the repo for s3fs-fuse-awscred-lib and added a line of logging, so I can see that my expiration milliseconds is the wrong value here: https://github.com/ggtakec/s3fs-fuse-awscred-lib/blob/master/awscred_func.cpp#L162. I am not sure why. If I decode my token, it has the right expiration timestamp.

kerem commented

Author

Owner

@sheromon commented on GitHub (Jun 28, 2023):

Ah, I think I know what my problem is. It seems like when AWS gives out credentials, the AWS_SESSION_TOKEN is not a JWT. I did a quick search on how to decode it and still don't know how after reading this StackOverflow thread: https://stackoverflow.com/questions/58140818/decoding-an-aws-session-token.

Anyway, I know that the AWS_SESSION_TOKEN I'm getting from MinIO's STS is a JWT, so that means the values I'm getting from AWS vs. MinIO are in different formats, and I can expect the standard AWS processing to produce incorrect values. Actually, I'm kind of surprised that the credentials worked at all and only the expiration seems to come out wrong.

I guess the solution for my problem is that we could write our own version of a cred lib that works with MinIO.

@sheromon commented on GitHub (Jun 28, 2023): Ah, I think I know what my problem is. It seems like when AWS gives out credentials, the `AWS_SESSION_TOKEN` is not a JWT. I did a quick search on how to decode it and still don't know how after reading this StackOverflow thread: https://stackoverflow.com/questions/58140818/decoding-an-aws-session-token. Anyway, I know that the `AWS_SESSION_TOKEN` I'm getting from MinIO's STS is a JWT, so that means the values I'm getting from AWS vs. MinIO are in different formats, and I can expect the standard AWS processing to produce incorrect values. Actually, I'm kind of surprised that the credentials worked at all and only the expiration seems to come out wrong. I guess the solution for my problem is that we could write our own version of a cred lib that works with MinIO.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Aug 28, 2023):

Could I know what's the conclusion for this thread? Does s3fs support refresh with AWS_SESSION_TOKEN with correct expiration date now? Or is it still WIP to support it?

@brucez-anyscale commented on GitHub (Aug 28, 2023): Could I know what's the conclusion for this thread? Does s3fs support refresh with `AWS_SESSION_TOKEN` with correct expiration date now? Or is it still WIP to support it?

kerem commented

Author

Owner

@sheromon commented on GitHub (Aug 29, 2023):

@brucez-anyscale, I believe that if you wanted to, you could use the s3fs-fuse-awscred-lib plugin to refresh your AWS credentials. However, you would need the plugin, as s3fs-fuse it self does not support this capability. There is a separate issue for built-in capability: #1068.

@sheromon commented on GitHub (Aug 29, 2023): @brucez-anyscale, I believe that if you wanted to, you could use the [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib) plugin to refresh your AWS credentials. However, you would need the plugin, as s3fs-fuse it self does not support this capability. There is a separate issue for built-in capability: #1068.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Aug 29, 2023):

@brucez-anyscale, I believe that if you wanted to, you could use the s3fs-fuse-awscred-lib plugin to refresh your AWS credentials. However, you would need the plugin, as s3fs-fuse it self does not support this capability. There is a separate issue for built-in capability: #1068.

Thanks!
So:

I need the plugin and mount with -o credlib=libs3fsawscred.so
I need to refresh the ${HOME}/.aws/credentials file myself periodically to provide validate tokens.

If I achieve both, the s3fs mount will always work without credential expiration?

@brucez-anyscale commented on GitHub (Aug 29, 2023): > @brucez-anyscale, I believe that if you wanted to, you could use the [s3fs-fuse-awscred-lib](https://github.com/ggtakec/s3fs-fuse-awscred-lib) plugin to refresh your AWS credentials. However, you would need the plugin, as s3fs-fuse it self does not support this capability. There is a separate issue for built-in capability: #1068. Thanks! So: 1. I need the plugin and mount with `-o credlib=libs3fsawscred.so` 2. I need to refresh the `${HOME}/.aws/credentials` file myself periodically to provide validate tokens. If I achieve both, the s3fs mount will always work without credential expiration?

kerem commented

Author

Owner

@sheromon commented on GitHub (Aug 29, 2023):

@brucez-anyscale, that sounds about right to me. I actually never got this fully working since what I am using is actually MinIO STS and not AWS STS, but in trying to figure out what was going wrong, I ended up digging into this code and ended up confident that it would work if I were using AWS STS. 😭

Looking back at the options I was running, I had -o dbglevel=info for more logging messages, -o credlib=/usr/lib/libs3fsawscred.so to use the plugin, -o credlib_opts=Debug for even more logging, and -o url={sts_addr} to set a custom URL for the STS endpoint.

@sheromon commented on GitHub (Aug 29, 2023): @brucez-anyscale, that sounds about right to me. I actually never got this fully working since what I am using is actually MinIO STS and not AWS STS, but in trying to figure out what was going wrong, I ended up digging into this code and ended up confident that it would work if I were using AWS STS. :sob: Looking back at the options I was running, I had `-o dbglevel=info` for more logging messages, `-o credlib=/usr/lib/libs3fsawscred.so` to use the plugin, `-o credlib_opts=Debug` for even more logging, and `-o url={sts_addr}` to set a custom URL for the STS endpoint.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Aug 29, 2023):

@brucez-anyscale, that sounds about right to me. I actually never got this fully working since what I am using is actually MinIO STS and not AWS STS, but in trying to figure out what was going wrong, I ended up digging into this code and ended up confident that it would work if I were using AWS STS. 😭

Looking back at the options I was running, I had -o dbglevel=info for more logging messages, -o credlib=/usr/lib/libs3fsawscred.so to use the plugin, -o credlib_opts=Debug for even more logging, and -o url={sts_addr} to set a custom URL for the STS endpoint.

Thank you very much!
I have setup the -o credlib=/usr/lib/libs3fsawscred.so, and I will keep an eye on how it works.
Will update back to this thread with any finding I have.

@brucez-anyscale commented on GitHub (Aug 29, 2023): > @brucez-anyscale, that sounds about right to me. I actually never got this fully working since what I am using is actually MinIO STS and not AWS STS, but in trying to figure out what was going wrong, I ended up digging into this code and ended up confident that it would work if I were using AWS STS. 😭 > > Looking back at the options I was running, I had `-o dbglevel=info` for more logging messages, `-o credlib=/usr/lib/libs3fsawscred.so` to use the plugin, `-o credlib_opts=Debug` for even more logging, and `-o url={sts_addr}` to set a custom URL for the STS endpoint. Thank you very much! I have setup the `-o credlib=/usr/lib/libs3fsawscred.so`, and I will keep an eye on how it works. Will update back to this thread with any finding I have.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Aug 31, 2023):

@sheromon
I did this:

Plugin and mount with -o credlib=libs3fsawscred.so
Refresh the ${HOME}/.aws/credentials file myself periodically to provide validate tokens.

But from curl dbg logs, I found:
2023-08-31T06:14:24.621Z [CURL DBG] > x-amz-content-sha256: {XXX}
2023-08-31T06:14:24.621Z [CURL DBG] > x-amz-date: {XXX}
2023-08-31T06:14:24.621Z [CURL DBG] > x-amz-security-token: {FOO}

The content of {FOO} does not update all the time. Even if the ${HOME}/.aws/credentials file is updated.

So I get
2023-08-31T06:14:24.628Z [CURL DBG] < HTTP/1.1 400 Bad Request
2023-08-31T06:14:24.629Z [ERR] curl.cpp:RequestPerform(2526): HEAD HTTP response code 400, returning EPERM.

Then if I access the s3fs mount folder, I got:
ls: reading directory '/mounts3fs': Input/output error

Anything wrong with my setup? Thanks!

@brucez-anyscale commented on GitHub (Aug 31, 2023): @sheromon I did this: - Plugin and mount with -o credlib=libs3fsawscred.so - Refresh the ${HOME}/.aws/credentials file myself periodically to provide validate tokens. But from curl dbg logs, I found: 2023-08-31T06:14:24.621Z [CURL DBG] > x-amz-content-sha256: {XXX} 2023-08-31T06:14:24.621Z [CURL DBG] > x-amz-date: {XXX} 2023-08-31T06:14:24.621Z [CURL DBG] > x-amz-security-token: {FOO} The content of {FOO} does not update all the time. Even if the ${HOME}/.aws/credentials file is updated. So I get 2023-08-31T06:14:24.628Z [CURL DBG] < HTTP/1.1 400 Bad Request 2023-08-31T06:14:24.629Z [ERR] curl.cpp:RequestPerform(2526): HEAD HTTP response code 400, returning EPERM. Then if I access the s3fs mount folder, I got: ls: reading directory '/mounts3fs': Input/output error Anything wrong with my setup? Thanks!

kerem commented

Author

Owner

@sheromon commented on GitHub (Aug 31, 2023):

@brucez-anyscale, ah sorry, my memory on this has been fuzzy since it's been a couple of months since I was looking at it. Trying to remember... I think you don't even need the file at ${HOME}/.aws/credentials. So the thing about what happens when you use the libs3fsawscred plugin is that the plugin itself isn't really looking for your credentials, it mainly just tells the AWS C++ SDK to go figure out your credentials situation. One thing that messed me up is the fact that I think if you have a ${HOME}/.aws/credentials file, then it just reads it and doesn't hit the STS endpoint, like it checks for the credentials file before asking STS. That was a problem for me, but I guess it sounds like the desired behavior for you? Anyway, the AWS C++ SDK checks for a bunch of different forms of the credentials in a certain order. I'm not sure if you have an EVs set locally, like AWS_SESSION_TOKEN. I had that accidentally set once and that was totally messing me up.

I'm not sure how you are populating ${HOME}/.aws/credentials. I had no way to get credentials other than by using STS. If you are in a similar situation, you can skip making the file, and the AWS C++ SDK should automatically get credentials from STS for you behind-the-scenes. I'm trying to think if using the credentials file should also work for automatic refreshing? I'm not sure.

I can offer the notes that I have that I found most helpful. One was that I found some good info on the s3fs-fuse FAQ page that I did not get from reading the main docs. And the other bit I have are these notes I took after inspecting the code.

s3fs-fuse calls the external credlib in S3fsCred::UpdateExtCredentials
UpdateExtCredentials gets called by S3fsCred::CheckIAMCredentialUpdate
CheckIAMCredentialUpdate is called in two places
- s3fs_check_service, which only runs when s3fs-fuse starts
- S3fsCurl::insertAuthHeaders, which is called by ‎S3fsCurl::RequestPerform, which gets called in several places

@sheromon commented on GitHub (Aug 31, 2023): @brucez-anyscale, ah sorry, my memory on this has been fuzzy since it's been a couple of months since I was looking at it. Trying to remember... I think you don't even need the file at `${HOME}/.aws/credentials`. So the thing about what happens when you use the `libs3fsawscred` plugin is that the plugin itself isn't really looking for your credentials, it mainly just tells the AWS C++ SDK to go figure out your credentials situation. One thing that messed me up is the fact that I think if you have a `${HOME}/.aws/credentials` file, then it just reads it and doesn't hit the STS endpoint, like it checks for the credentials file before asking STS. That was a problem for me, but I guess it sounds like the desired behavior for you? Anyway, the AWS C++ SDK checks for a bunch of different forms of the credentials in a certain order. I'm not sure if you have an EVs set locally, like `AWS_SESSION_TOKEN`. I had that accidentally set once and that was totally messing me up. I'm not sure how you are populating `${HOME}/.aws/credentials`. I had no way to get credentials other than by using STS. If you are in a similar situation, you can skip making the file, and the AWS C++ SDK should automatically get credentials from STS for you behind-the-scenes. I'm trying to think if using the `credentials` file should also work for automatic refreshing? I'm not sure. I can offer the notes that I have that I found most helpful. One was that I found some good info on the [s3fs-fuse FAQ](https://github.com/s3fs-fuse/s3fs-fuse/wiki/FAQ) page that I did not get from reading the main docs. And the other bit I have are these notes I took after inspecting the code. - s3fs-fuse calls the external credlib in [S3fsCred::UpdateExtCredentials](https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/s3fs_cred.cpp#L1353) - UpdateExtCredentials gets called by [S3fsCred::CheckIAMCredentialUpdate](https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/s3fs_cred.cpp#L1159) - CheckIAMCredentialUpdate is called in two places - [s3fs_check_service](https://github.com/s3fs-fuse/s3fs-fuse/blob/7c9cf843168ce2a870b20a954b823f3c8100ec5d/src/s3fs.cpp#L4350C20-L4350C44), which only runs when s3fs-fuse starts - [S3fsCurl::insertAuthHeaders](https://github.com/s3fs-fuse/s3fs-fuse/blob/7c9cf843168ce2a870b20a954b823f3c8100ec5d/src/curl.cpp#L2954), which is called by ‎[S3fsCurl::RequestPerform](https://github.com/s3fs-fuse/s3fs-fuse/blob/7c9cf843168ce2a870b20a954b823f3c8100ec5d/src/curl.cpp#L2479), which gets called in several places

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Aug 31, 2023):

Thanks very much for the info! Let me try it out.
I will delete ${HOME}/.aws/credentials and see if AWS C++ SDK can just fetch the credentials.

@brucez-anyscale commented on GitHub (Aug 31, 2023): Thanks very much for the info! Let me try it out. I will delete `${HOME}/.aws/credentials` and see if AWS C++ SDK can just fetch the credentials.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Sep 1, 2023):

@sheromon I delete ${HOME}/.aws/credentials and just run mount with -o credlib=libs3fsawscred.so
It auto mounts the s3 bucket for me without asking for credentials.

But after a while like serval hours:
I still get this error

ls: reading directory '/mounts3fs': Input/output error

Maybe I need to do something special for the credential refreshment?
Thanks

@brucez-anyscale commented on GitHub (Sep 1, 2023): @sheromon I delete `${HOME}/.aws/credentials` and just run mount with -o credlib=libs3fsawscred.so It auto mounts the s3 bucket for me without asking for credentials. But after a while like serval hours: I still get this error ``` ls: reading directory '/mounts3fs': Input/output error ``` Maybe I need to do something special for the credential refreshment? Thanks

kerem commented

Author

Owner

@sheromon commented on GitHub (Sep 1, 2023):

@brucez-anyscale Yeah, it sounds like your credentials are expiring, and you are not getting a new set. I'm out of tips, though. What helped me when I was trying to figure out how this works was looking at the log messages. I mentioned the options I used to get debug logging messages previously, so yeah, that's all I got. Good luck!

@sheromon commented on GitHub (Sep 1, 2023): @brucez-anyscale Yeah, it sounds like your credentials are expiring, and you are not getting a new set. I'm out of tips, though. What helped me when I was trying to figure out how this works was looking at the log messages. I mentioned the options I used to get debug logging messages previously, so yeah, that's all I got. Good luck!

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Sep 2, 2023):

thanks very much. I will check the logs!

@brucez-anyscale commented on GitHub (Sep 2, 2023): thanks very much. I will check the logs!

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Sep 2, 2023):

I guess this could be the root cause. It gets HTTP/1.1 404 Not Found. Seems the url is //. I will debug more.

2023-09-02T00:04:48.932Z [ERR] s3fs_xml.cpp:get_base_exp(114): marker_xp->nodesetval is empty.

2023-09-02T00:04:48.904Z [CURL DBG] * Found bundle for host  [serially]
2023-09-02T00:04:48.904Z [CURL DBG] * Can not multiplex, even if we wanted to!
2023-09-02T00:04:48.904Z [CURL DBG] * Re-using existing connection! (#1) with host 
2023-09-02T00:04:48.904Z [CURL DBG] * Connected to
2023-09-02T00:04:48.904Z [CURL DBG] > HEAD // HTTP/1.1
2023-09-02T00:04:48.904Z [CURL DBG] > Host: 
2023-09-02T00:04:48.904Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash 48548f08; OpenSSL)
2023-09-02T00:04:48.904Z [CURL DBG] > Accept: */*
2023-09-02T00:04:48.904Z [CURL DBG] > Authorization: 
2023-09-02T00:04:48.904Z [CURL DBG] > x-amz-content-sha256: 
2023-09-02T00:04:48.904Z [CURL DBG] > x-amz-date: 
2023-09-02T00:04:48.904Z [CURL DBG] > x-amz-security-token: 
2023-09-02T00:04:48.904Z [CURL DBG] >
2023-09-02T00:04:48.913Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-09-02T00:04:48.913Z [CURL DBG] < HTTP/1.1 404 Not Found
2023-09-02T00:04:48.913Z [CURL DBG] < x-amz-request-id: 
2023-09-02T00:04:48.913Z [CURL DBG] < x-amz-id-2: 
2023-09-02T00:04:48.913Z [CURL DBG] < Content-Type: application/xml
2023-09-02T00:04:48.913Z [CURL DBG] < Date: Sat, 02 Sep 2023 00:04:48 GMT
2023-09-02T00:04:48.913Z [CURL DBG] < Server: AmazonS3

after debugging more, seems this is a cache miss error.

@brucez-anyscale commented on GitHub (Sep 2, 2023): I guess this could be the root cause. It gets `HTTP/1.1 404 Not Found`. Seems the url is `//`. I will debug more. ``` 2023-09-02T00:04:48.932Z [ERR] s3fs_xml.cpp:get_base_exp(114): marker_xp->nodesetval is empty. 2023-09-02T00:04:48.904Z [CURL DBG] * Found bundle for host [serially] 2023-09-02T00:04:48.904Z [CURL DBG] * Can not multiplex, even if we wanted to! 2023-09-02T00:04:48.904Z [CURL DBG] * Re-using existing connection! (#1) with host 2023-09-02T00:04:48.904Z [CURL DBG] * Connected to 2023-09-02T00:04:48.904Z [CURL DBG] > HEAD // HTTP/1.1 2023-09-02T00:04:48.904Z [CURL DBG] > Host: 2023-09-02T00:04:48.904Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash 48548f08; OpenSSL) 2023-09-02T00:04:48.904Z [CURL DBG] > Accept: */* 2023-09-02T00:04:48.904Z [CURL DBG] > Authorization: 2023-09-02T00:04:48.904Z [CURL DBG] > x-amz-content-sha256: 2023-09-02T00:04:48.904Z [CURL DBG] > x-amz-date: 2023-09-02T00:04:48.904Z [CURL DBG] > x-amz-security-token: 2023-09-02T00:04:48.904Z [CURL DBG] > 2023-09-02T00:04:48.913Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-09-02T00:04:48.913Z [CURL DBG] < HTTP/1.1 404 Not Found 2023-09-02T00:04:48.913Z [CURL DBG] < x-amz-request-id: 2023-09-02T00:04:48.913Z [CURL DBG] < x-amz-id-2: 2023-09-02T00:04:48.913Z [CURL DBG] < Content-Type: application/xml 2023-09-02T00:04:48.913Z [CURL DBG] < Date: Sat, 02 Sep 2023 00:04:48 GMT 2023-09-02T00:04:48.913Z [CURL DBG] < Server: AmazonS3 ``` after debugging more, seems this is a cache miss error.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Sep 2, 2023):

Now I think the root cause is this: s3fsawscred think the token expires long time later.

[s3fsawscred] : expiration    = 2262-04-11T16:47:16Z

@brucez-anyscale commented on GitHub (Sep 2, 2023): Now I think the root cause is this: s3fsawscred think the token expires long time later. ``` [s3fsawscred] : expiration = 2262-04-11T16:47:16Z ```

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Sep 8, 2023):

Update: After fixing the token expiration issue in our STS side code, the libs3fsawscred plugin works for us. Thanks!

@brucez-anyscale commented on GitHub (Sep 8, 2023): Update: After fixing the token expiration issue in our STS side code, the libs3fsawscred plugin works for us. Thanks!

kerem commented

Author

Owner

@skumar19822 commented on GitHub (Dec 19, 2023):

Update: After fixing the token expiration issue in our STS side code, the libs3fsawscred plugin works for us. Thanks!

Thanks @brucez-anyscale. I was also facing the same issue [s3fsawscred] : expiration = 2262-04-11T16:47:16Z.
Kindly help me out what changes you made in STS code that would be really helpful.

@skumar19822 commented on GitHub (Dec 19, 2023): > Update: After fixing the token expiration issue in our STS side code, the libs3fsawscred plugin works for us. Thanks! Thanks @brucez-anyscale. I was also facing the same issue [s3fsawscred] : expiration = 2262-04-11T16:47:16Z. Kindly help me out what changes you made in STS code that would be really helpful.

kerem commented

Author

Owner

@brucez-anyscale commented on GitHub (Dec 20, 2023):

Update: After fixing the token expiration issue in our STS side code, the libs3fsawscred plugin works for us. Thanks!

Thanks @brucez-anyscale. I was also facing the same issue [s3fsawscred] : expiration = 2262-04-11T16:47:16Z. Kindly help me out what changes you made in STS code that would be really helpful.

I previously connect to AWS directly to fetch the token. And it does not work.
So I set up a node as a server in AWS which has token itself, and let my library talk to my AWS node.

So it is not a general fix.

@brucez-anyscale commented on GitHub (Dec 20, 2023): > > Update: After fixing the token expiration issue in our STS side code, the libs3fsawscred plugin works for us. Thanks! > > Thanks @brucez-anyscale. I was also facing the same issue [s3fsawscred] : expiration = 2262-04-11T16:47:16Z. Kindly help me out what changes you made in STS code that would be really helpful. I previously connect to AWS directly to fetch the token. And it does not work. So I set up a node as a server in AWS which has token itself, and let my library talk to my AWS node. So it is not a general fix.

kerem commented

Author

Owner

@ggtakec commented on GitHub (Mar 3, 2024):

@sheromon @brucez-anyscale Sorry for my very late reply.
I've updated the libs3fsawscred plugin( libs3fsawscred #10 ) slightly and added TokenPeriodSecond option.

The reason why the expiration date cannot be set correctly in libs3fsawscred is because it cannot be obtained.
Since we were mainly thinking about AWS, there was no processing to decode MinIO's JWT format, and we were unable to set an expiration date.
In any case, if the expiration date is reached, there is no logic to re-acquire the token, so you may need to reset the credential externally.

If only the expiry date is required, it can be given as an option in the PR above. (but the initial behavior of it is incomplete)

It would be helpful if you could post about the STS login process and expiration date using libs3fsawscred in the libs3fsawscred issue.

@ggtakec commented on GitHub (Mar 3, 2024): @sheromon @brucez-anyscale Sorry for my very late reply. I've updated the libs3fsawscred plugin( [libs3fsawscred #10](https://github.com/ggtakec/s3fs-fuse-awscred-lib/pull/10) ) slightly and added `TokenPeriodSecond` option. The reason why the expiration date cannot be set correctly in libs3fsawscred is because it cannot be obtained. Since we were mainly thinking about AWS, there was no processing to decode MinIO's JWT format, and we were unable to set an expiration date. In any case, if the expiration date is reached, there is no logic to re-acquire the token, so you may need to reset the credential externally. If only the expiry date is required, it can be given as an option in the PR above. (but the initial behavior of it is incomplete) It would be helpful if you could post about the STS login process and expiration date using libs3fsawscred in the libs3fsawscred issue.

kerem commented

Author

Owner

@sheromon commented on GitHub (Mar 3, 2024):

@ggtakec, thank you for the update! Honestly, I haven't been thinking about this recently and don't remember many of the details, but what you explained is useful for me. I may come back to trying to use s3fs-fuse together with libs3fsawscred in the future, but not sure when, and I appreciate your work.

I am not sure which GitHub issue you were referring to here, but I think I don't remember enough to say anything meaningful at this point...

It would be helpful if you could post about the STS login process and expiration date using libs3fsawscred in the libs3fsawscred issue.

@sheromon commented on GitHub (Mar 3, 2024): @ggtakec, thank you for the update! Honestly, I haven't been thinking about this recently and don't remember many of the details, but what you explained is useful for me. I may come back to trying to use s3fs-fuse together with libs3fsawscred in the future, but not sure when, and I appreciate your work. I am not sure which GitHub issue you were referring to here, but I think I don't remember enough to say anything meaningful at this point... > It would be helpful if you could post about the STS login process and expiration date using libs3fsawscred in the libs3fsawscred issue.

kerem commented