[GH-ISSUE #2391] Fails to mount volume after reboot of IMDSv2 instance #1175

New issue

Open

opened 2026-03-04 01:51:56 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:51:56 +03:00

Owner

Originally created by @mosaic-nick on GitHub (Dec 15, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2391

Additional Information

Version of s3fs being used (`s3fs --version`)

V1.93 (commit:unknown)

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

2.9.2

Kernel information (`uname -r`)

5.10.201-191.748.amzn2.x86_64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3⭕amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

How to run s3fs, if applicable

s3fs#:/srcdata /var/opt/mosaic/srcdata fuse uid=,gid=**********,allow_other,mp_umask=277,umask=277,use_cache=/tmp,iam_role=auto,url=https://s3.eu-west-1.amazonaws.com,endpoint=eu-west-1,dbglevel=debug 0 0

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: s3fs version 1.93(unknown) : s3fs -o rw,uid=995,gid=1002,allow_other,mp_umask=277,umask=277,use_cache=/tmp,iam_role=auto,url=https://s3.eu-west-1.amazonaws.com,endpoint=eu-west-1,dbglevel=debug,dev,suid ************:/srcdata /var/opt/mosaic/srcdata
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [DBG]
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]:    PROC(uid=0, gid=0) - MountPoint(uid=995, gid=1002, mode=40705)Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: Loaded mime information from /etc/mime.types
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2577]: init v1.93(commit:unknown) with OpenSSL, credential-library(built-in)
Dec 15 10:37:02 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2620): ### CURLE_COULDNT_CONNECT
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: ### retrying...
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]:      Retry request. [type=-1][url=http://169.254.169.254/latest/api/token][path=]
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: Failed to reset handle and internal data for retrying.
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: curl.cpp:GetIAMv2ApiToken(3055): Error(-5) occurred, could not get IAMv2 api token.
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: s3fs_cred.cpp:GetIAMCredentialsURL(409): AWS IMDSv2 token retrieval failed: -5
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]:      Get IAM Role name
Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2620): ### CURLE_COULDNT_CONNECT
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: ### retrying...
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]:      Retry request. [type=14][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/][path=]
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2598): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>#012<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"#012#011"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">#012<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">#012 <head>#012  <title>401 - Unauthorized</title>#012 </head>#012 <body>#012  <h1>401 - Unauthorized</h1>#012 </body>#012</html>
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: curl.cpp:GetIAMRoleFromMetaData(3199): Error(-5) occurred, could not get IAM role name from meta data.
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: s3fs.cpp:s3fs_init(4296): could not load IAM role name from meta data.
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: s3fs.cpp:s3fs_exit_fuseloop(4277): Exiting FUSE event loop due to errors
Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: destroy

Details about issue

Upon rebooting an instance using IMDSv2, the s3fs mount is not mounted successfully. From the logs, it looks like it tries to get an IMDSv2 token, but curl fails to connect, suggesting that the IMDS endpoint is not available at that point. It then seems to retry, but still can't connect, and then errors, but tries to continue to retrieve the IAM role anyway, which then fails due to a lack of the IMDSv2 token.

Logging on to the instance and running "mount -a" then successfully mounts the volume.

Originally created by @mosaic-nick on GitHub (Dec 15, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2391  ### Additional Information #### Version of s3fs being used (`s3fs --version`) V1.93 (commit:unknown) #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 2.9.2 #### Kernel information (`uname -r`) 5.10.201-191.748.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" SUPPORT_END="2025-06-30" #### How to run s3fs, if applicable s3fs#**********:/srcdata /var/opt/mosaic/srcdata fuse uid=**********,gid=**********,allow_other,mp_umask=277,umask=277,use_cache=/tmp,iam_role=auto,url=https://s3.eu-west-1.amazonaws.com,endpoint=eu-west-1,dbglevel=debug 0 0 #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) ``` Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: s3fs version 1.93(unknown) : s3fs -o rw,uid=995,gid=1002,allow_other,mp_umask=277,umask=277,use_cache=/tmp,iam_role=auto,url=https://s3.eu-west-1.amazonaws.com,endpoint=eu-west-1,dbglevel=debug,dev,suid ************:/srcdata /var/opt/mosaic/srcdata Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [DBG] Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: PROC(uid=0, gid=0) - MountPoint(uid=995, gid=1002, mode=40705)Dec 15 10:37:02 ip-10-7-4-122 s3fs[2572]: Loaded mime information from /etc/mime.types Dec 15 10:37:02 ip-10-7-4-122 s3fs[2577]: init v1.93(commit:unknown) with OpenSSL, credential-library(built-in) Dec 15 10:37:02 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2620): ### CURLE_COULDNT_CONNECT Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: ### retrying... Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: Retry request. [type=-1][url=http://169.254.169.254/latest/api/token][path=] Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: Failed to reset handle and internal data for retrying. Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: curl.cpp:GetIAMv2ApiToken(3055): Error(-5) occurred, could not get IAMv2 api token. Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: s3fs_cred.cpp:GetIAMCredentialsURL(409): AWS IMDSv2 token retrieval failed: -5 Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: Get IAM Role name Dec 15 10:37:06 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2620): ### CURLE_COULDNT_CONNECT Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: ### retrying... Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: Retry request. [type=14][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/][path=] Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: curl.cpp:RequestPerform(2598): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>#012<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"#012#011"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">#012<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">#012 <head>#012 <title>401 - Unauthorized</title>#012 </head>#012 <body>#012 <h1>401 - Unauthorized</h1>#012 </body>#012</html> Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: curl.cpp:GetIAMRoleFromMetaData(3199): Error(-5) occurred, could not get IAM role name from meta data. Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: s3fs.cpp:s3fs_init(4296): could not load IAM role name from meta data. Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: s3fs.cpp:s3fs_exit_fuseloop(4277): Exiting FUSE event loop due to errors Dec 15 10:37:10 ip-10-7-4-122 s3fs[2577]: destroy ``` ### Details about issue Upon rebooting an instance using IMDSv2, the s3fs mount is not mounted successfully. From the logs, it looks like it tries to get an IMDSv2 token, but curl fails to connect, suggesting that the IMDS endpoint is not available at that point. It then seems to retry, but still can't connect, and then errors, but tries to continue to retrieve the IAM role anyway, which then fails due to a lack of the IMDSv2 token. Logging on to the instance and running "mount -a" then successfully mounts the volume.

kerem commented

2026-03-04 01:51:57 +03:00

Author

Owner

@mosaic-nick commented on GitHub (Dec 15, 2023):

It seems that the issue is that it is trying to mount before the network service is running. I have been able to work around this by setting the following options on the mount in fstab:

_netdev,x-systemd.after=network-online.target

@mosaic-nick commented on GitHub (Dec 15, 2023): It seems that the issue is that it is trying to mount before the network service is running. I have been able to work around this by setting the following options on the mount in fstab: _netdev,x-systemd.after=network-online.target