[GH-ISSUE #651 ] Where to put aws_session_token information #371

Author

Owner

@ifnull commented on GitHub (Aug 17, 2018):

Ultimately it needs to be passed in the request header as x-amz-security-token. Which seems to be set in S3fsCurl::AWSAccessToken. I attempted to add support by adding the AWS SessionToken to /etc/passwd-s3fs as a key/value pair but found that it was going to be more work that just adding it to parse_passwd_file(). I don’t really know C++ so I’m not having much luck. This would be a great feature if someone want to add it. For now, I'm going to have to use aws s3 sync and keep a cached copy of everything locally.

Using Temporary Security Credentials to Request Access to AWS Resources
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html

@ifnull commented on GitHub (Aug 17, 2018): Ultimately it needs to be passed in the request header as `x-amz-security-token`. Which seems to be set in `S3fsCurl::AWSAccessToken`. I attempted to add support by adding the AWS `SessionToken` to `/etc/passwd-s3fs` as a key/value pair but found that it was going to be more work that just adding it to `parse_passwd_file()`. I don’t really know C++ so I’m not having much luck. This would be a great feature if someone want to add it. For now, I'm going to have to use `aws s3 sync` and keep a cached copy of everything locally. Using Temporary Security Credentials to Request Access to AWS Resources https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 11, 2019):

Please test with -o use_session_token and reopen if this does not address your use case.

@gaul commented on GitHub (Jul 11, 2019): Please test with `-o use_session_token` and reopen if this does not address your use case.

kerem commented

Author

Owner

@ekarmazin commented on GitHub (Jul 24, 2019):

s3fs gaves me:
fuse: unknown option `use_session_token'

@ekarmazin commented on GitHub (Jul 24, 2019): s3fs gaves me: fuse: unknown option `use_session_token'

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 24, 2019):

Which version did you test with? No release version yet supports this; you have to compile from master.

@gaul commented on GitHub (Jul 24, 2019): Which version did you test with? No release version yet supports this; you have to compile from master.

kerem commented

Author

Owner

@ekarmazin commented on GitHub (Jul 24, 2019):

Oh ok, I've used 1.85, let me try master and get back to here.

@ekarmazin commented on GitHub (Jul 24, 2019): Oh ok, I've used 1.85, let me try master and get back to here.

kerem commented

Author

Owner

@ekarmazin commented on GitHub (Jul 24, 2019):

s3fs built from master branch, see latest commit:

Amazon Simple Storage Service File System V1.85 (commit:80972aa) with OpenSSL
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

s3fs error:

[CRT] s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF]
[INF] s3fs.cpp:set_mountpoint_attribute(4379): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
fuse: unknown option `use_session_token'

@ekarmazin commented on GitHub (Jul 24, 2019): s3fs built from master branch, see latest commit: Amazon Simple Storage Service File System V1.85 (commit:80972aa) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. s3fs error: [CRT] s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4379): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) fuse: unknown option `use_session_token'

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 24, 2019):

I believe use_session_token goes in $HOME/.aws/credentials.

@gaul commented on GitHub (Jul 24, 2019): I believe `use_session_token` goes in `$HOME/.aws/credentials`.

kerem commented

Author

Owner

@ekarmazin commented on GitHub (Jul 24, 2019):

No, ~/.aws/credentials has different naming:

aws_access_key_id = KEY_GOES_HERE
aws_secret_access_key = SECRET_GOES_HERE
aws_session_token = TOKEN_GOES_HERE

I've tried to pass 'aws_session_token' in s3fs options but no luck, same error: fuse: unknown option `aws_session_token'

weird, but I see that feature in cpp listing:

if(profile != aws_profile){
return EXIT_FAILURE;
}
if (session_token.empty()) {
if (is_use_session_token) {
S3FS_PRN_EXIT("AWS session token was expected but wasn't provided in aws/credentials file for profile: %s.", aws_profile.c_str());
return EXIT_FAILURE;
}

But I am getting an error about 'no such option' instead of missing token or something.

@ekarmazin commented on GitHub (Jul 24, 2019): No, ~/.aws/credentials has different naming: aws_access_key_id = KEY_GOES_HERE aws_secret_access_key = SECRET_GOES_HERE aws_session_token = TOKEN_GOES_HERE I've tried to pass 'aws_session_token' in s3fs options but no luck, same error: fuse: unknown option `aws_session_token' weird, but I see that feature in cpp listing: if(profile != aws_profile){ return EXIT_FAILURE; } if (session_token.empty()) { if (is_use_session_token) { S3FS_PRN_EXIT("AWS session token was expected but wasn't provided in aws/credentials file for profile: %s.", aws_profile.c_str()); return EXIT_FAILURE; } But I am getting an error about 'no such option' instead of missing token or something.

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 24, 2019):

Sorry, $HOME/.aws/credentials should contain aws_session_token and the command-line flags should contain -o use_session_token.

@gaul commented on GitHub (Jul 24, 2019): Sorry, `$HOME/.aws/credentials` should contain `aws_session_token` and the command-line flags should contain `-o use_session_token`.

kerem commented

Author

Owner

@ekarmazin commented on GitHub (Jul 24, 2019):

Thats what I did, and got error not about incorrect or missing token but:

fuse: unknown option `use_session_token'

while specifying cli flag for s3fs built form master branch.

@ekarmazin commented on GitHub (Jul 24, 2019): Thats what I did, and got error not about incorrect or missing token but: fuse: unknown option `use_session_token' while specifying cli flag for s3fs built form master branch.

kerem commented

Author

Owner

@gaul commented on GitHub (Jul 24, 2019):

How exactly do you involve s3fs? I see the following:

$ src/s3fs BUCKET PATH -f -o use_session_token
s3fs: AWS session token was expected but wasn't provided in aws/credentials file for profile: default.
s3fs: could not determine how to establish security credentials.

@gaul commented on GitHub (Jul 24, 2019): How exactly do you involve s3fs? I see the following: ``` $ src/s3fs BUCKET PATH -f -o use_session_token s3fs: AWS session token was expected but wasn't provided in aws/credentials file for profile: default. s3fs: could not determine how to establish security credentials. ```

kerem commented

Author

Owner

@juliogonzalez commented on GitHub (Jul 24, 2019):

Wild suggestion:

@ekarmazin keep in mind that if you are using /etc/fstab and you have both 1.85 and master build installed at the same time, most probably the system is using s3fs 1.85 and not master.

@juliogonzalez commented on GitHub (Jul 24, 2019): Wild suggestion: @ekarmazin keep in mind that if you are using `/etc/fstab` and you have both 1.85 and master build installed at the same time, most probably the system is using s3fs 1.85 and not master.

kerem commented

Author

Owner

@ekarmazin commented on GitHub (Jul 24, 2019):

I am using docker containers and each build is a fresh one, so no version overlapping.

Here is my configs and how I run s3fs (git clone and make were done at /tmp):

/tmp/s3fs-fuse/src/s3fs MY_BUCKET_HERE /mnt -o use_session_token -o allow_other -o use_cache=/tmp -o profile="PROFILE_FORM_AWS_CREDS" -o use_path_request_style -o dbglevel=info -f -o curldbg

cat ~/.aws/credentials
[ACCOUNT-PROFILENAME]
aws_access_key_id     = ASOMEKEYHERE723B
aws_secret_access_key = wxENSOMESECURITYKEYHEREhw67dQW
aws_session_token     = FQoGLONGLONGTOKENHEREhTkFYNEYt

See commit #, the latest from master, so it was build from master branch:

s3fs --version
Amazon Simple Storage Service File System V1.85 (commit:80972aa) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Error I am getting:

[CRT] s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF]
[INF]     s3fs.cpp:set_mountpoint_attribute(4379): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
fuse: unknown option `use_session_token'

Point me, what I am doing wrong. Here is how I compiled from src:

cd /tmp && \
git clone https://github.com/s3fs-fuse/s3fs-fuse.git && \
cd s3fs-fuse && \
./autogen.sh && \
./configure  && \
make && \
make install

fuse version: 2.9.x

@ekarmazin commented on GitHub (Jul 24, 2019): I am using docker containers and each build is a fresh one, so no version overlapping. **Here is my configs and how I run s3fs (git clone and make were done at /tmp):** /tmp/s3fs-fuse/src/s3fs MY_BUCKET_HERE /mnt -o use_session_token -o allow_other -o use_cache=/tmp -o profile="PROFILE_FORM_AWS_CREDS" -o use_path_request_style -o dbglevel=info -f -o curldbg ``` cat ~/.aws/credentials [ACCOUNT-PROFILENAME] aws_access_key_id = ASOMEKEYHERE723B aws_secret_access_key = wxENSOMESECURITYKEYHEREhw67dQW aws_session_token = FQoGLONGLONGTOKENHEREhTkFYNEYt ``` **See commit #, the latest from master, so it was build from master branch:** ``` s3fs --version Amazon Simple Storage Service File System V1.85 (commit:80972aa) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` **Error I am getting:** ``` [CRT] s3fs.cpp:set_s3fs_log_level(297): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4379): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) fuse: unknown option `use_session_token' ``` **Point me, what I am doing wrong. Here is how I compiled from src:** cd /tmp && \ git clone https://github.com/s3fs-fuse/s3fs-fuse.git && \ cd s3fs-fuse && \ ./autogen.sh && \ ./configure && \ make && \ make install fuse version: 2.9.x

kerem commented

Author

Owner

@saurabhdas commented on GitHub (Nov 27, 2019):

Can this issue be re-opened? I'm facing the same problem: I get:

fuse: unknown option `use_session_token'

when I invoke s3fs with -o use_session_token.

@saurabhdas commented on GitHub (Nov 27, 2019): Can this issue be re-opened? I'm facing the same problem: I get: ```fuse: unknown option `use_session_token'``` when I invoke s3fs with `-o use_session_token`.

kerem commented

Author

Owner

@gaul commented on GitHub (Nov 27, 2019):

Which version of s3fs do you use? This option only exists in master, not 1.85.

@gaul commented on GitHub (Nov 27, 2019): Which version of s3fs do you use? This option only exists in master, not 1.85.

kerem commented

Author

Owner

@saurabhdas commented on GitHub (Dec 3, 2019):

For the s3fs installation, I am following the same procedure as @ekarmazin – compiling from source.

I later discovered that doing this works:

export AWSACCESSKEYID=${AWS_ACCESS_KEY_ID}
export AWSSECRETACCESSKEY=${AWS_SECRET_ACCESS_KEY}
export AWSSESSIONTOKEN=${AWS_SESSION_TOKEN}
s3fs -o retries=20 $S3_BUCKET $MOUNT_PATH

The -o use_session_token option causes a failure as documented above. Perhaps this is correct/intentional, however something in the documentation led me to believe that I needed to use -o use_session_token to enable this behaviour.

@saurabhdas commented on GitHub (Dec 3, 2019): For the s3fs installation, I am following the same procedure as @ekarmazin – compiling from source. I later discovered that doing this works: ``` export AWSACCESSKEYID=${AWS_ACCESS_KEY_ID} export AWSSECRETACCESSKEY=${AWS_SECRET_ACCESS_KEY} export AWSSESSIONTOKEN=${AWS_SESSION_TOKEN} s3fs -o retries=20 $S3_BUCKET $MOUNT_PATH ``` The `-o use_session_token` option causes a failure as documented above. Perhaps this is correct/intentional, however something in the documentation led me to believe that I needed to use `-o use_session_token` to enable this behaviour.

kerem commented

Author

Owner

@wwhurley commented on GitHub (Jan 15, 2020):

I'm having the same issue. I'm pulling master from the repo, using Amazon Simple Storage Service File System V1.85 (commit:e0712f4) with OpenSSL with Fuse 2.9.7 and when running the command s3fs [BUCKET]:[PATH] [MOUNT] -f -o nonempty -o use_session_token I'm getting the error fuse: unknown option use_session_token'`

@wwhurley commented on GitHub (Jan 15, 2020): I'm having the same issue. I'm pulling master from the repo, using `Amazon Simple Storage Service File System V1.85 (commit:e0712f4) with OpenSSL` with Fuse 2.9.7 and when running the command `s3fs [BUCKET]:[PATH] [MOUNT] -f -o nonempty -o use_session_token` I'm getting the error `fuse: unknown option `use_session_token'`

kerem commented

Author

Owner

@gaul commented on GitHub (Sep 10, 2020):

Could someone test with the latest master which includes a flag parsing fix?

@gaul commented on GitHub (Sep 10, 2020): Could someone test with the latest master which includes a flag parsing fix?

kerem commented

Author

Owner

@llitz commented on GitHub (Sep 16, 2020):

Working fine for me with c58c91f

@llitz commented on GitHub (Sep 16, 2020): Working fine for me with c58c91f

kerem commented

Author

Owner

@tiilikainen commented on GitHub (Oct 14, 2020):

Still does not work for me. I'm using the brew install version which bills itself as 1.87 with FUSE 2.9.7. However, --version displays commit:unknown.

@tiilikainen commented on GitHub (Oct 14, 2020): Still does not work for me. I'm using the `brew install` version which bills itself as 1.87 with FUSE 2.9.7. However, `--version` displays `commit:unknown`.

kerem commented