[GH-ISSUE #2192] s3fs not working with EKS #1115

New issue

Closed

opened 2026-03-04 01:51:31 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:51:31 +03:00

Owner

Originally created by @ricosega on GitHub (Jun 21, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2192

Additional Information

When running in EKS we found many issues. We have to give container privileged: true this causes the s3fs (do not know why) to access the node iam role which interferes with the role provided through the serviceaccount. So if we run the command with -o iam_role="auto" it takes the node role but if we set the role name instead we receive a 404 error when trying to get instance metadata as you will see below.

Version of s3fs being used (`s3fs --version`)

V1.92 (commit:7c9cf843168ce2a870b20a954b823f3c8100ec5d)

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

2.9.9

Kernel information (`uname -r`)

5.10.179-168.710.amzn2.x86_64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

Arch Linux

How to run s3fs, if applicable

s3fs BUCKET_NAME:/PATH_FOLDER /mnt -o iam_role="auto" -o dbglevel=info -o url="https://s3.eu-central-1.amazonaws.com/" -o endpoint="eu-central-1" -o curldbg -o allow_other -o profile="default" -f

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

[root@s3fs-good-arch .aws]# s3fs $COMPANY:/$PATH_TYPE /mnt -o iam_role="deleteme" -o dbglevel=info -o url="https://s3.eu-central-1.amazonaws.com/" -o endpoint="eu-central-1" -o curldbg -o allow_other -o profile="default" -f
2023-06-21T15:47:35.048Z [INF] s3fs version 1.92(7c9cf84) : s3fs -o iam_role=deleteme -o dbglevel=info -o url=https://s3.eu-central-1.amazonaws.com/ -o endpoint=eu-central-1 -o curldbg -o allow_other -o profile=default -f xcompany01:/shared /mnt
2023-06-21T15:47:35.049Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [INF] 
2023-06-21T15:47:35.049Z [INF]     s3fs.cpp:set_mountpoint_attribute(4479): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777)
2023-06-21T15:47:35.049Z [INF] s3fs_util.cpp:compare_sysname(421): system name is Linux
2023-06-21T15:47:35.049Z [WAN] curl.cpp:InitMimeType(403): Could not find mime.types files, you have to create file(/etc/mime.types) or specify mime option for existing mime.types file.
2023-06-21T15:47:35.049Z [WAN] s3fs.cpp:main(5527): Missing MIME types prevents setting Content-Type on uploaded objects.
2023-06-21T15:47:35.049Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(78): The path to cache top dir is empty, thus not need to check permission.
2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do.
2023-06-21T15:47:35.049Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-06-21T15:47:35.049Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-06-21T15:47:35.049Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-06-21T15:47:35.049Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-06-21T15:47:35.049Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-06-21T15:47:35.056Z [INF] s3fs.cpp:s3fs_init(4210): init v1.92(commit:7c9cf84) with OpenSSL, credential-library(built-in)
2023-06-21T15:47:35.056Z [INF] s3fs.cpp:s3fs_check_service(4347): check services.
2023-06-21T15:47:35.056Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1150): IAM Access Token refreshing...
2023-06-21T15:47:35.056Z [INF]       s3fs_cred.cpp:GetIAMCredentialsURL(372): [IAM role=deleteme]
2023-06-21T15:47:35.056Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-06-21T15:47:35.056Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2023-06-21T15:47:35.056Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2023-06-21T15:47:35.056Z [CURL DBG] > Host: 169.254.169.254
2023-06-21T15:47:35.056Z [CURL DBG] > User-Agent: s3fs/1.92 (commit hash 7c9cf84; OpenSSL)
2023-06-21T15:47:35.056Z [CURL DBG] > Accept: */*
2023-06-21T15:47:35.057Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2023-06-21T15:47:35.057Z [CURL DBG] > Content-Length: 0
2023-06-21T15:47:35.057Z [CURL DBG] > 
2023-06-21T15:47:35.057Z [CURL DBG] < HTTP/1.1 200 OK
2023-06-21T15:47:35.057Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-06-21T15:47:35.057Z [CURL DBG] < Content-Length: 56
2023-06-21T15:47:35.057Z [CURL DBG] < Date: Wed, 21 Jun 2023 15:47:35 GMT
2023-06-21T15:47:35.057Z [CURL DBG] < Server: EC2ws
2023-06-21T15:47:35.057Z [CURL DBG] < Connection: close
2023-06-21T15:47:35.057Z [CURL DBG] < Content-Type: text/plain
2023-06-21T15:47:35.057Z [CURL DBG] < 
2023-06-21T15:47:35.057Z [CURL DBG] * Closing connection 0
2023-06-21T15:47:35.057Z [INF]       curl.cpp:RequestPerform(2519): HTTP response code 200
2023-06-21T15:47:35.057Z [INF]       s3fs_cred.cpp:SetIAMv2APIToken(445): Setting AWS IMDSv2 API token to AQAEAKfG2bGuW4qIvGAn4GfhHW8-SrUNy0_TzSccLyykH3ciKO21RA==
2023-06-21T15:47:35.057Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2023-06-21T15:47:35.057Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-06-21T15:47:35.057Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1)
2023-06-21T15:47:35.058Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/deleteme HTTP/1.1
2023-06-21T15:47:35.058Z [CURL DBG] > Host: 169.254.169.254
2023-06-21T15:47:35.058Z [CURL DBG] > User-Agent: s3fs/1.92 (commit hash 7c9cf84; OpenSSL)
2023-06-21T15:47:35.058Z [CURL DBG] > Accept: */*
2023-06-21T15:47:35.058Z [CURL DBG] > X-aws-ec2-metadata-token: AQAEAKfG2bGuW4qIvGAn4GfhHW8-SrUNy0_TzSccLyykH3ciKO21RA==
2023-06-21T15:47:35.058Z [CURL DBG] > 
2023-06-21T15:47:35.058Z [CURL DBG] < HTTP/1.1 404 Not Found
2023-06-21T15:47:35.058Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-06-21T15:47:35.058Z [CURL DBG] < Content-Type: text/html
2023-06-21T15:47:35.058Z [CURL DBG] < Content-Length: 339
2023-06-21T15:47:35.058Z [CURL DBG] < Date: Wed, 21 Jun 2023 15:47:35 GMT
2023-06-21T15:47:35.058Z [CURL DBG] < Server: EC2ws
2023-06-21T15:47:35.058Z [CURL DBG] < Connection: close
2023-06-21T15:47:35.058Z [CURL DBG] < 
2023-06-21T15:47:35.058Z [CURL DBG] * Closing connection 1
2023-06-21T15:47:35.058Z [INF]       curl.cpp:RequestPerform(2571): HTTP response code 404 was returned, returning ENOENT
2023-06-21T15:47:35.058Z [ERR] curl.cpp:GetIAMCredentials(3140): Error(-2) occurred, could not get IAM role name.
2023-06-21T15:47:35.058Z [ERR] s3fs_cred.cpp:CheckIAMCredentialUpdate(1155): Access Token refresh by built-in failed
2023-06-21T15:47:35.058Z [CRT] s3fs.cpp:s3fs_check_service(4351): Failed to initialize IAM credential.
2023-06-21T15:47:35.058Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(4200): Exiting FUSE event loop due to errors

2023-06-21T15:47:35.059Z [INF] s3fs.cpp:s3fs_destroy(4254): destroy

Details about issue

How can we do to make this work in EKS but using IAM roles attached to the container?

Originally created by @ricosega on GitHub (Jun 21, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2192  ### Additional Information When running in EKS we found many issues. We have to give container `privileged: true` this causes the s3fs (do not know why) to access the node iam role which interferes with the role provided through the serviceaccount. So if we run the command with `-o iam_role="auto"` it takes the node role but if we set the role name instead we receive a 404 error when trying to get instance metadata as you will see below. #### Version of s3fs being used (`s3fs --version`) V1.92 (commit:7c9cf843168ce2a870b20a954b823f3c8100ec5d) #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 2.9.9 #### Kernel information (`uname -r`) 5.10.179-168.710.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) Arch Linux #### How to run s3fs, if applicable ``` s3fs BUCKET_NAME:/PATH_FOLDER /mnt -o iam_role="auto" -o dbglevel=info -o url="https://s3.eu-central-1.amazonaws.com/" -o endpoint="eu-central-1" -o curldbg -o allow_other -o profile="default" -f ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` [root@s3fs-good-arch .aws]# s3fs $COMPANY:/$PATH_TYPE /mnt -o iam_role="deleteme" -o dbglevel=info -o url="https://s3.eu-central-1.amazonaws.com/" -o endpoint="eu-central-1" -o curldbg -o allow_other -o profile="default" -f 2023-06-21T15:47:35.048Z [INF] s3fs version 1.92(7c9cf84) : s3fs -o iam_role=deleteme -o dbglevel=info -o url=https://s3.eu-central-1.amazonaws.com/ -o endpoint=eu-central-1 -o curldbg -o allow_other -o profile=default -f xcompany01:/shared /mnt 2023-06-21T15:47:35.049Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [INF] 2023-06-21T15:47:35.049Z [INF] s3fs.cpp:set_mountpoint_attribute(4479): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777) 2023-06-21T15:47:35.049Z [INF] s3fs_util.cpp:compare_sysname(421): system name is Linux 2023-06-21T15:47:35.049Z [WAN] curl.cpp:InitMimeType(403): Could not find mime.types files, you have to create file(/etc/mime.types) or specify mime option for existing mime.types file. 2023-06-21T15:47:35.049Z [WAN] s3fs.cpp:main(5527): Missing MIME types prevents setting Content-Type on uploaded objects. 2023-06-21T15:47:35.049Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(78): The path to cache top dir is empty, thus not need to check permission. 2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do. 2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-06-21T15:47:35.049Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-06-21T15:47:35.056Z [INF] s3fs.cpp:s3fs_init(4210): init v1.92(commit:7c9cf84) with OpenSSL, credential-library(built-in) 2023-06-21T15:47:35.056Z [INF] s3fs.cpp:s3fs_check_service(4347): check services. 2023-06-21T15:47:35.056Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1150): IAM Access Token refreshing... 2023-06-21T15:47:35.056Z [INF] s3fs_cred.cpp:GetIAMCredentialsURL(372): [IAM role=deleteme] 2023-06-21T15:47:35.056Z [CURL DBG] * Trying 169.254.169.254:80... 2023-06-21T15:47:35.056Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2023-06-21T15:47:35.056Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2023-06-21T15:47:35.056Z [CURL DBG] > Host: 169.254.169.254 2023-06-21T15:47:35.056Z [CURL DBG] > User-Agent: s3fs/1.92 (commit hash 7c9cf84; OpenSSL) 2023-06-21T15:47:35.056Z [CURL DBG] > Accept: */* 2023-06-21T15:47:35.057Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2023-06-21T15:47:35.057Z [CURL DBG] > Content-Length: 0 2023-06-21T15:47:35.057Z [CURL DBG] > 2023-06-21T15:47:35.057Z [CURL DBG] < HTTP/1.1 200 OK 2023-06-21T15:47:35.057Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-06-21T15:47:35.057Z [CURL DBG] < Content-Length: 56 2023-06-21T15:47:35.057Z [CURL DBG] < Date: Wed, 21 Jun 2023 15:47:35 GMT 2023-06-21T15:47:35.057Z [CURL DBG] < Server: EC2ws 2023-06-21T15:47:35.057Z [CURL DBG] < Connection: close 2023-06-21T15:47:35.057Z [CURL DBG] < Content-Type: text/plain 2023-06-21T15:47:35.057Z [CURL DBG] < 2023-06-21T15:47:35.057Z [CURL DBG] * Closing connection 0 2023-06-21T15:47:35.057Z [INF] curl.cpp:RequestPerform(2519): HTTP response code 200 2023-06-21T15:47:35.057Z [INF] s3fs_cred.cpp:SetIAMv2APIToken(445): Setting AWS IMDSv2 API token to AQAEAKfG2bGuW4qIvGAn4GfhHW8-SrUNy0_TzSccLyykH3ciKO21RA== 2023-06-21T15:47:35.057Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache 2023-06-21T15:47:35.057Z [CURL DBG] * Trying 169.254.169.254:80... 2023-06-21T15:47:35.057Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1) 2023-06-21T15:47:35.058Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/deleteme HTTP/1.1 2023-06-21T15:47:35.058Z [CURL DBG] > Host: 169.254.169.254 2023-06-21T15:47:35.058Z [CURL DBG] > User-Agent: s3fs/1.92 (commit hash 7c9cf84; OpenSSL) 2023-06-21T15:47:35.058Z [CURL DBG] > Accept: */* 2023-06-21T15:47:35.058Z [CURL DBG] > X-aws-ec2-metadata-token: AQAEAKfG2bGuW4qIvGAn4GfhHW8-SrUNy0_TzSccLyykH3ciKO21RA== 2023-06-21T15:47:35.058Z [CURL DBG] > 2023-06-21T15:47:35.058Z [CURL DBG] < HTTP/1.1 404 Not Found 2023-06-21T15:47:35.058Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-06-21T15:47:35.058Z [CURL DBG] < Content-Type: text/html 2023-06-21T15:47:35.058Z [CURL DBG] < Content-Length: 339 2023-06-21T15:47:35.058Z [CURL DBG] < Date: Wed, 21 Jun 2023 15:47:35 GMT 2023-06-21T15:47:35.058Z [CURL DBG] < Server: EC2ws 2023-06-21T15:47:35.058Z [CURL DBG] < Connection: close 2023-06-21T15:47:35.058Z [CURL DBG] < 2023-06-21T15:47:35.058Z [CURL DBG] * Closing connection 1 2023-06-21T15:47:35.058Z [INF] curl.cpp:RequestPerform(2571): HTTP response code 404 was returned, returning ENOENT 2023-06-21T15:47:35.058Z [ERR] curl.cpp:GetIAMCredentials(3140): Error(-2) occurred, could not get IAM role name. 2023-06-21T15:47:35.058Z [ERR] s3fs_cred.cpp:CheckIAMCredentialUpdate(1155): Access Token refresh by built-in failed 2023-06-21T15:47:35.058Z [CRT] s3fs.cpp:s3fs_check_service(4351): Failed to initialize IAM credential. 2023-06-21T15:47:35.058Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(4200): Exiting FUSE event loop due to errors 2023-06-21T15:47:35.059Z [INF] s3fs.cpp:s3fs_destroy(4254): destroy ``` ### Details about issue  How can we do to make this work in EKS but using IAM roles attached to the container?