starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1605] The use of AWS Metadata Version 2 with EC2 Instances breaks S3FS mounts #841

New issue
Open
opened 2026-03-04 01:49:16 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:49:16 +03:00
Owner
Copy link

Originally created by @eelbrink on GitHub (Mar 16, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1605

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

s3fs --version
Amazon Simple Storage Service File System V1.86 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 104
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Architecture: amd64
Version: 2.9.7-1ubuntu1
Depends: libc6 (>= 2.14), libfuse2 (= 2.9.7-1ubuntu1), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
/etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
export a virtual filesystem to the Linux kernel. It also aims to provide a
secure method for non privileged users to create and mount their own filesystem
implementations.
Homepage: https://github.com/libfuse/libfuse/wiki
Original-Maintainer: Laszlo Boszormenyi (GCS) gcs@debian.org

Kernel information (uname -r)

5.4.0-1025-aws

GNU/Linux Distribution, if applicable (cat /etc/os-release)

cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

s3fs command line used, if applicable

/etc/fstab entry, if applicable

# Mounting the S3 [data-d1-c1] bucket
data-d1-c1: /data/d1/c1 fuse.s3fs umask=007,gid=3000,_netdev,allow_other,iam_role=auto,use_sse=kmsid:arn:aws:kms:eu-west-1:**************:key/************************* 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Mar 16 19:18:57 c1 s3fs[10599]: init v1.86(commit:unknown) with OpenSSL
Mar 16 19:18:57 c1 s3fs[10599]: s3fs.cpp:s3fs_init(3506): could not load IAM role name from meta data.

Details about issue

When we implemented AWS EC2 Metadata Version 2 for our EC2 instances due to security reasons, s3f3 does not connect to s3 bucket anymore.

The below code shows how Metadata version 2 is enabled.

 "MetadataOptions": {
   "State": "applied",
    "HttpTokens": "required",
     "HttpPutResponseHopLimit": 1,
     "HttpEndpoint": "enabled"
},

It complains with the following error:

s3fs[10380]: s3fs.cpp:s3fs_init(3506): could not load IAM role name from meta data.

Originally created by @eelbrink on GitHub (Mar 16, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1605 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) _s3fs --version Amazon Simple Storage Service File System V1.86 (commit:unknown) with OpenSSL_ #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 104 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Architecture: amd64 Version: 2.9.7-1ubuntu1 Depends: libc6 (>= 2.14), libfuse2 (= 2.9.7-1ubuntu1), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Homepage: https://github.com/libfuse/libfuse/wiki Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>_ #### Kernel information (uname -r) _5.4.0-1025-aws_ #### GNU/Linux Distribution, if applicable (cat /etc/os-release) _cat /etc/os-release NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic_ #### s3fs command line used, if applicable ``` ``` #### /etc/fstab entry, if applicable ``` # Mounting the S3 [data-d1-c1] bucket data-d1-c1: /data/d1/c1 fuse.s3fs umask=007,gid=3000,_netdev,allow_other,iam_role=auto,use_sse=kmsid:arn:aws:kms:eu-west-1:**************:key/************************* 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` Mar 16 19:18:57 c1 s3fs[10599]: init v1.86(commit:unknown) with OpenSSL Mar 16 19:18:57 c1 s3fs[10599]: s3fs.cpp:s3fs_init(3506): could not load IAM role name from meta data. ``` ### Details about issue When we implemented AWS EC2 Metadata Version 2 for our EC2 instances due to security reasons, s3f3 does not connect to s3 bucket anymore. The below code shows how Metadata version 2 is enabled. ``` "MetadataOptions": { "State": "applied", "HttpTokens": "required", "HttpPutResponseHopLimit": 1, "HttpEndpoint": "enabled" }, ``` It complains with the following error: > s3fs[10380]: s3fs.cpp:s3fs_init(3506): could not load IAM role name from meta data.
kerem commented 2026-03-04 01:49:17 +03:00
Author
Owner
Copy link

@thenickdude commented on GitHub (Mar 17, 2021):

Support for IMDSv2 was only added in 1.88, you're using the older 1.86. Update to the latest release.

<!-- gh-comment-id:801494697 --> @thenickdude commented on GitHub (Mar 17, 2021): Support for IMDSv2 was only added in 1.88, you're using the older 1.86. Update to the latest release.
kerem commented 2026-03-04 01:49:17 +03:00
Author
Owner
Copy link

@eelbrink commented on GitHub (Mar 24, 2021):

Thanks @thenickdude, I will update the once that are still using an older version. However, how would this actually work? Does S3FS request the secure token itself? or do we have to request the token first and then run S3FS with use_session_token in order for it to work?

<!-- gh-comment-id:805633542 --> @eelbrink commented on GitHub (Mar 24, 2021): Thanks @thenickdude, I will update the once that are still using an older version. However, how would this actually work? Does S3FS request the secure token itself? or do we have to request the token first and then run S3FS with `use_session_token` in order for it to work?
kerem commented 2026-03-04 01:49:17 +03:00
Author
Owner
Copy link

@kornicameister commented on GitHub (Nov 2, 2021):

Any update on question from @eelbrink? @eelbrink perhaps you've been able to determine how this work? Most specifically how that is going to work with /etc/fstab?

<!-- gh-comment-id:957173292 --> @kornicameister commented on GitHub (Nov 2, 2021): Any update on question from @eelbrink? @eelbrink perhaps you've been able to determine how this work? Most specifically how that is going to work with /etc/fstab?
kerem commented 2026-03-04 01:49:17 +03:00
Author
Owner
Copy link

@kornicameister commented on GitHub (Nov 2, 2021):

If I try to use s3fs like so: s3fs some-bucket ./mount-point -o use_cache=/tmp -o iam_role=MyRole -o use_session_token -f -d I get following output and bucket does not mount:

2021-11-02T07:51:48.968Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF]
2021-11-02T07:51:48.968Z [INF]     s3fs.cpp:set_mountpoint_attribute(4094): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775)
2021-11-02T07:51:48.970Z [INF] curl.cpp:InitMimeType(434): Loaded mime information from /etc/mime.types
2021-11-02T07:51:48.973Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:unknown) with OpenSSL
2021-11-02T07:51:48.973Z [INF] s3fs.cpp:s3fs_check_service(3516): check services.
2021-11-02T07:51:48.974Z [INF] curl.cpp:CheckIAMCredentialUpdate(1770): IAM Access Token refreshing...
2021-11-02T07:51:48.974Z [INF]       curl.cpp:GetIAMCredentials(2822): [IAM role=MyRole]
2021-11-02T07:51:48.975Z [ERR] curl.cpp:RequestPerform(2394): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>417 - Expectation Failed</title>
 </head>
 <body>
  <h1>417 - Expectation Failed</h1>
 </body>
</html>

2021-11-02T07:51:48.976Z [ERR] curl.cpp:GetIAMCredentials(2859): AWS IMDSv2 token retrieval failed: -5
2021-11-02T07:51:48.977Z [ERR] curl.cpp:RequestPerform(2394): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>

2021-11-02T07:51:48.978Z [ERR] curl.cpp:CheckIAMCredentialUpdate(1774): IAM Access Token refresh failed
2021-11-02T07:51:48.978Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2021-11-02T07:51:48.978Z [CRT] s3fs.cpp:s3fs_check_service(3520): Failed to check IAM role name(AYSecureEC2Access).
2021-11-02T07:51:48.978Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors

2021-11-02T07:51:48.993Z [INF] s3fs.cpp:s3fs_destroy(3440): destroy

and I do run s3fs --version: v1.90

<!-- gh-comment-id:957177439 --> @kornicameister commented on GitHub (Nov 2, 2021): If I try to use `s3fs` like so: `s3fs some-bucket ./mount-point -o use_cache=/tmp -o iam_role=MyRole -o use_session_token -f -d` I get following output and bucket does not mount: ``` 2021-11-02T07:51:48.968Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF] 2021-11-02T07:51:48.968Z [INF] s3fs.cpp:set_mountpoint_attribute(4094): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775) 2021-11-02T07:51:48.970Z [INF] curl.cpp:InitMimeType(434): Loaded mime information from /etc/mime.types 2021-11-02T07:51:48.973Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:unknown) with OpenSSL 2021-11-02T07:51:48.973Z [INF] s3fs.cpp:s3fs_check_service(3516): check services. 2021-11-02T07:51:48.974Z [INF] curl.cpp:CheckIAMCredentialUpdate(1770): IAM Access Token refreshing... 2021-11-02T07:51:48.974Z [INF] curl.cpp:GetIAMCredentials(2822): [IAM role=MyRole] 2021-11-02T07:51:48.975Z [ERR] curl.cpp:RequestPerform(2394): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>417 - Expectation Failed</title> </head> <body> <h1>417 - Expectation Failed</h1> </body> </html> 2021-11-02T07:51:48.976Z [ERR] curl.cpp:GetIAMCredentials(2859): AWS IMDSv2 token retrieval failed: -5 2021-11-02T07:51:48.977Z [ERR] curl.cpp:RequestPerform(2394): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>401 - Unauthorized</title> </head> <body> <h1>401 - Unauthorized</h1> </body> </html> 2021-11-02T07:51:48.978Z [ERR] curl.cpp:CheckIAMCredentialUpdate(1774): IAM Access Token refresh failed 2021-11-02T07:51:48.978Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler 2021-11-02T07:51:48.978Z [CRT] s3fs.cpp:s3fs_check_service(3520): Failed to check IAM role name(AYSecureEC2Access). 2021-11-02T07:51:48.978Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors 2021-11-02T07:51:48.993Z [INF] s3fs.cpp:s3fs_destroy(3440): destroy ``` and I do run `s3fs --version: v1.90`
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#841
No description provided.