starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2029] S3FS hanging on Ubuntu 22.04 on AWS EC2 instance #1018

New issue
Closed
opened 2026-03-04 01:50:41 +03:00 by kerem · 10 comments
kerem commented 2026-03-04 01:50:41 +03:00
Owner
Copy link

Originally created by @jpedrick on GitHub (Sep 8, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2029

Additional Information

Linux Distribution: Ubuntu 22.04 stock AWS AMI as of 2022-08-08

Version of s3fs being used (s3fs --version)

V9.1 compiled from source

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Fuse Version: 3.10.5

Kernel information (uname -r)

_command result: 5.15.0-1019-aws

GNU/Linux Distribution, if applicable (cat /etc/os-release)

_command result:

PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

s3fs command line used, if applicable

 s3fs my-bucket /mnt/my-bucket/ -o iam_role=auto -d -f -s -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.ap-east-1.amazonaws.com/

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

2022-09-08T22:35:29.227Z [INF] s3fs version 1.91(unknown) : s3fs -o iam_role=auto -d -f -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.ap-east-1.amazonaws.com/ -s my-bucket /mnt/my-bucket/
2022-09-08T22:35:29.228Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF] 
2022-09-08T22:35:29.228Z [INF]     s3fs.cpp:set_mountpoint_attribute(3627): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
2022-09-08T22:35:29.230Z [INF] curl.cpp:InitMimeType(424): Loaded mime information from /etc/mime.types
2022-09-08T22:35:29.230Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(80): The path to cache top dir is empty, thus not need to check permission.
2022-09-08T22:35:29.230Z [INF] s3fs.cpp:s3fs_init(3393): init v1.91(commit:unknown) with OpenSSL
2022-09-08T22:35:29.231Z [CURL DBG] *   Trying 169.254.169.254:80...
2022-09-08T22:35:29.231Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2022-09-08T22:35:29.231Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2022-09-08T22:35:29.231Z [CURL DBG] > Host: 169.254.169.254
2022-09-08T22:35:29.231Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash unknown; OpenSSL)
2022-09-08T22:35:29.231Z [CURL DBG] > Accept: */*
2022-09-08T22:35:29.231Z [CURL DBG] > Transfer-Encoding: chunked
2022-09-08T22:35:29.231Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2022-09-08T22:35:29.231Z [CURL DBG] >

Details about issue

After the above output the program seems to just hang.

my-bucket is a placeholder for the actual s3 bucket I am trying to access.

Originally created by @jpedrick on GitHub (Sep 8, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2029 ### Additional Information Linux Distribution: Ubuntu 22.04 stock AWS AMI as of 2022-08-08 #### Version of s3fs being used (s3fs --version) V9.1 compiled from source #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Fuse Version: 3.10.5 #### Kernel information (uname -r) _command result: 5.15.0-1019-aws #### GNU/Linux Distribution, if applicable (cat /etc/os-release) _command result: ``` PRETTY_NAME="Ubuntu 22.04.1 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.1 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` #### s3fs command line used, if applicable ``` s3fs my-bucket /mnt/my-bucket/ -o iam_role=auto -d -f -s -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.ap-east-1.amazonaws.com/ ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` 2022-09-08T22:35:29.227Z [INF] s3fs version 1.91(unknown) : s3fs -o iam_role=auto -d -f -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.ap-east-1.amazonaws.com/ -s my-bucket /mnt/my-bucket/ 2022-09-08T22:35:29.228Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF] 2022-09-08T22:35:29.228Z [INF] s3fs.cpp:set_mountpoint_attribute(3627): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) 2022-09-08T22:35:29.230Z [INF] curl.cpp:InitMimeType(424): Loaded mime information from /etc/mime.types 2022-09-08T22:35:29.230Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(80): The path to cache top dir is empty, thus not need to check permission. 2022-09-08T22:35:29.230Z [INF] s3fs.cpp:s3fs_init(3393): init v1.91(commit:unknown) with OpenSSL 2022-09-08T22:35:29.231Z [CURL DBG] * Trying 169.254.169.254:80... 2022-09-08T22:35:29.231Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2022-09-08T22:35:29.231Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2022-09-08T22:35:29.231Z [CURL DBG] > Host: 169.254.169.254 2022-09-08T22:35:29.231Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash unknown; OpenSSL) 2022-09-08T22:35:29.231Z [CURL DBG] > Accept: */* 2022-09-08T22:35:29.231Z [CURL DBG] > Transfer-Encoding: chunked 2022-09-08T22:35:29.231Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2022-09-08T22:35:29.231Z [CURL DBG] > ``` ### Details about issue After the above output the program seems to just hang. my-bucket is a placeholder for the actual s3 bucket I am trying to access.
kerem closed this issue 2026-03-04 01:50:42 +03:00
kerem commented 2026-03-04 01:50:42 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Sep 24, 2022):

You run by specifying url=https://s3.ap-east-1.amazonaws.com on the command line, but it seems to be accessing the instance metadata URL(169.254.169.254 and port 80).
Normally, I think s3fs would access the host as s3.ap-east-1.amazonaws.com.
Please check if you have set anything such as hosts or proxy or etc.

<!-- gh-comment-id:1256903392 --> @ggtakec commented on GitHub (Sep 24, 2022): You run by specifying `url=https://s3.ap-east-1.amazonaws.com` on the command line, but it seems to be accessing the instance metadata URL(`169.254.169.254` and port 80). Normally, I think s3fs would access the host as `s3.ap-east-1.amazonaws.com`. Please check if you have set anything such as hosts or proxy or etc.
kerem commented 2026-03-04 01:50:42 +03:00
Author
Owner
Copy link

@porelli commented on GitHub (Dec 5, 2022):

The instance metadata url is to retrieve the token and then the credentials. After retrieving the credentials it will connect to the specified S3 endpoint.

I did encounter the same problem using Debian 11, I tried the s3fs-fuse package both in the stable and testing repositories and also the latest release checked out of git.

The interesting part is that the AWS CLI works and also invoking the IMDSv2 manually doesn't seem to behave any differently. However, for some reasons, s3fs-fuse hangs while trying to retrieve the same set of credentials in the same way when running on Debian or Ubuntu.

The very same command and settings worked without any issue on another Amazon Linux sharing the same VPC, security group and role.

<!-- gh-comment-id:1338129344 --> @porelli commented on GitHub (Dec 5, 2022): The instance metadata url is to retrieve the token and then the credentials. After retrieving the credentials it will connect to the specified S3 endpoint. I did encounter the same problem using Debian 11, I tried the s3fs-fuse package both in the stable and testing repositories and also the latest release checked out of git. The interesting part is that the AWS CLI works and also invoking the IMDSv2 manually doesn't seem to behave any differently. However, for some reasons, s3fs-fuse hangs while trying to retrieve the same set of credentials in the same way when running on Debian or Ubuntu. The very same command and settings worked without any issue on another Amazon Linux sharing the same VPC, security group and role.
kerem commented 2026-03-04 01:50:42 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 12, 2023):

@porelli I'm sorry for my fault about not noticing your iam_role option and metadata url.
And I was able to reproduce the problem you pointed out, so I'll investigate.

<!-- gh-comment-id:1380400950 --> @ggtakec commented on GitHub (Jan 12, 2023): @porelli I'm sorry for my fault about not noticing your `iam_role` option and metadata url. And I was able to reproduce the problem you pointed out, so I'll investigate.
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 12, 2023):

@porelli
It seems that the instance you are using cannot use IMDSv2, so you can avoid the problem by setting it to use IMDSv1.

Specifying iam_role will try IMDSv2 first and fall back to IMDSv1 if that fails. However, this phenomenon occurs because it does not support IMDSv2.

So you can specify imdsv1only option to avoid it.
I think this option will solve your problem.
Please try it.

<!-- gh-comment-id:1380461670 --> @ggtakec commented on GitHub (Jan 12, 2023): @porelli It seems that the instance you are using cannot use `IMDSv2`, so you can avoid the problem by setting it to use `IMDSv1`. Specifying `iam_role` will try `IMDSv2` first and fall back to `IMDSv1` if that fails. However, this phenomenon occurs because it does not support `IMDSv2`. So you can specify `imdsv1only` option to avoid it. I think this option will solve your problem. Please try it.
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@porelli commented on GitHub (Jan 16, 2023):

Thank you for checking!
If it was, I'm not sure why the Debian instance had IMDSv2 disabled. Surely it should have had at least the IMDSv1 enabled but I do not have the instance anymore to re-try.

You are 100% correct, with the "-o imdsv1only" I can successfully mount the bucket. Altough, I'm not able to see the entire content. I suspect this might be because it was created manuall and not using using s3fs.

Regarding IMDSv2, I collected some more data that might be helpful for further investigation.
I just tried again with the latest code in the git repo and I can still reproduce the same problem on a new instance that definitely has both IMDSv1 and IMDSv2 enabled:
IMDSv1:

$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups
services/
system

IMDSv2:

~/s3fs-fuse/src$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0  14000      0 --:--:-- --:--:-- --:--:-- 14000
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.74.0
> Accept: */*
> X-aws-ec2-metadata-token: <<REDACTED-TOKEN>>
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Content-Type: text/plain
< Accept-Ranges: none
< Last-Modified: Mon, 16 Jan 2023 16:42:46 GMT
< Content-Length: 325
< Date: Mon, 16 Jan 2023 16:57:34 GMT
< Server: EC2ws
< Connection: close
< 
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
identity-credentials/
instance-action
instance-id
instance-life-cycle
instance-type
local-hostname
local-ipv4
mac
metrics/
network/
placement/
profile
public-hostname
public-ipv4
public-keys/
reservation-id
security-groups
services/
* Closing connection 0
system

IMDSv2 with wrong token:

$ TOKEN=fake && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.74.0
> Accept: */*
> X-aws-ec2-metadata-token: fake
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Content-Length: 0
< Date: Mon, 16 Jan 2023 17:00:51 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
< 
* Closing connection 0

S3FS hangs as @jpedrick reported in this ticket:

$ sudo ./s3fs <<REDACTED-BUCKET-NAME>> /tmp/bucket/ -o iam_role=auto -d -f -s -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.us-east-2.amazonaws.com/
2023-01-16T17:03:40.218Z [INF] s3fs version 1.91(16bc449) : s3fs -o iam_role=auto -d -f -s -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.us-east-2.amazonaws.com/ <<REDACTED-BUCKET-NAME>> /tmp/bucket/
2023-01-16T17:03:40.219Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [INF] 
2023-01-16T17:03:40.219Z [INF]     s3fs.cpp:set_mountpoint_attribute(4191): PROC(uid=0, gid=0) - MountPoint(uid=1000, gid=1000, mode=40755)
2023-01-16T17:03:40.221Z [INF] curl.cpp:InitMimeType(428): Loaded mime information from /etc/mime.types
2023-01-16T17:03:40.222Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(78): The path to cache top dir is empty, thus not need to check permission.
2023-01-16T17:03:40.222Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do.
2023-01-16T17:03:40.224Z [CRT] s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function.
2023-01-16T17:03:40.224Z [INF] s3fs.cpp:s3fs_init(3914): init v1.91(commit:16bc449) with OpenSSL, credential-library(built-in)
2023-01-16T17:03:40.224Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-16T17:03:40.225Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2023-01-16T17:03:40.225Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2023-01-16T17:03:40.225Z [CURL DBG] > Host: 169.254.169.254
2023-01-16T17:03:40.225Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-16T17:03:40.225Z [CURL DBG] > Accept: */*
2023-01-16T17:03:40.225Z [CURL DBG] > Transfer-Encoding: chunked
2023-01-16T17:03:40.225Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2023-01-16T17:03:40.225Z [CURL DBG] > 
2023-01-16T17:03:40.225Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-16T17:03:40.225Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-16T17:03:40.225Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-16T17:03:40.225Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-16T17:03:40.225Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.

If I press ctrl+c it continues but then hangs again:

^C2023-01-16T17:05:07.680Z [CURL DBG] * Signaling end of chunked upload via terminating chunk.
2023-01-16T17:05:07.681Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-16T17:05:07.681Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-16T17:05:07.681Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-01-16T17:05:07.681Z [CURL DBG] < Content-Length: 56
2023-01-16T17:05:07.681Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:05:07 GMT
2023-01-16T17:05:07.681Z [CURL DBG] < Server: EC2ws
2023-01-16T17:05:07.681Z [CURL DBG] < Connection: close
2023-01-16T17:05:07.681Z [CURL DBG] < Content-Type: text/plain
2023-01-16T17:05:07.681Z [CURL DBG] < 
2023-01-16T17:05:07.681Z [CURL DBG] * Closing connection 0
2023-01-16T17:05:07.681Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-16T17:05:07.681Z [INF]       s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to <<REDACTED-TOKEN>>
2023-01-16T17:05:07.681Z [INF]       curl.cpp:GetIAMRoleFromMetaData(3006): Get IAM Role name
2023-01-16T17:05:07.681Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-16T17:05:07.681Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1)
2023-01-16T17:05:07.681Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
2023-01-16T17:05:07.681Z [CURL DBG] > Host: 169.254.169.254
2023-01-16T17:05:07.681Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-16T17:05:07.681Z [CURL DBG] > Accept: */*
2023-01-16T17:05:07.681Z [CURL DBG] > X-aws-ec2-metadata-token: <<REDACTED-TOKEN>>
2023-01-16T17:05:07.681Z [CURL DBG] > 
2023-01-16T17:05:07.682Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-16T17:05:07.682Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-16T17:05:07.682Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-01-16T17:05:07.682Z [CURL DBG] < Content-Type: text/plain
2023-01-16T17:05:07.682Z [CURL DBG] < Accept-Ranges: none
2023-01-16T17:05:07.682Z [CURL DBG] < Last-Modified: Mon, 16 Jan 2023 16:42:46 GMT
2023-01-16T17:05:07.682Z [CURL DBG] < Content-Length: 19
2023-01-16T17:05:07.682Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:05:07 GMT
2023-01-16T17:05:07.682Z [CURL DBG] < Server: EC2ws
2023-01-16T17:05:07.682Z [CURL DBG] < Connection: close
2023-01-16T17:05:07.682Z [CURL DBG] < 
2023-01-16T17:05:07.682Z [CURL DBG] * Closing connection 1
2023-01-16T17:05:07.682Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-16T17:05:07.682Z [INF]       s3fs_cred.cpp:SetIAMRoleFromMetaData(582): IAM role name response = "<<REDACTED-BUCKET-NAME>>"
2023-01-16T17:05:07.682Z [INF] s3fs_cred.cpp:LoadIAMRoleFromMetaData(543): loaded IAM role name = <<REDACTED-BUCKET-NAME>>
2023-01-16T17:05:07.682Z [INF] s3fs.cpp:s3fs_check_service(4058): check services.
2023-01-16T17:05:07.682Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1152): IAM Access Token refreshing...
2023-01-16T17:05:07.682Z [INF]       s3fs_cred.cpp:GetIAMCredentialsURL(374): [IAM role=<<REDACTED-BUCKET-NAME>>]
2023-01-16T17:05:07.682Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2023-01-16T17:05:07.682Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-16T17:05:07.682Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#2)
2023-01-16T17:05:07.682Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2023-01-16T17:05:07.682Z [CURL DBG] > Host: 169.254.169.254
2023-01-16T17:05:07.682Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-16T17:05:07.682Z [CURL DBG] > Accept: */*
2023-01-16T17:05:07.682Z [CURL DBG] > Transfer-Encoding: chunked
2023-01-16T17:05:07.682Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2023-01-16T17:05:07.682Z [CURL DBG] >

At this point, if I try to access the folder from another terminal, the shell hangs there. Pressing CTRL+C once more in the original shell:

^C2023-01-16T17:06:36.304Z [CURL DBG] * Signaling end of chunked upload via terminating chunk.
2023-01-16T17:06:36.304Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-16T17:06:36.304Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-16T17:06:36.304Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-01-16T17:06:36.305Z [CURL DBG] < Content-Length: 56
2023-01-16T17:06:36.305Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:36 GMT
2023-01-16T17:06:36.305Z [CURL DBG] < Server: EC2ws
2023-01-16T17:06:36.305Z [CURL DBG] < Connection: close
2023-01-16T17:06:36.305Z [CURL DBG] < Content-Type: text/plain
2023-01-16T17:06:36.305Z [CURL DBG] < 
2023-01-16T17:06:36.305Z [CURL DBG] * Closing connection 2
2023-01-16T17:06:36.305Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-16T17:06:36.305Z [INF]       s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to <<REDACTED-TOKEN>>
2023-01-16T17:06:36.305Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-16T17:06:36.305Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#3)
2023-01-16T17:06:36.305Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/<<REDACTED-BUCKET-NAME>> HTTP/1.1
2023-01-16T17:06:36.305Z [CURL DBG] > Host: 169.254.169.254
2023-01-16T17:06:36.305Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-16T17:06:36.305Z [CURL DBG] > Accept: */*
2023-01-16T17:06:36.305Z [CURL DBG] > X-aws-ec2-metadata-token: <<REDACTED-TOKEN>>
2023-01-16T17:06:36.305Z [CURL DBG] > 
2023-01-16T17:06:36.305Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-16T17:06:36.305Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-16T17:06:36.305Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-01-16T17:06:36.305Z [CURL DBG] < Content-Type: text/plain
2023-01-16T17:06:36.305Z [CURL DBG] < Accept-Ranges: none
2023-01-16T17:06:36.305Z [CURL DBG] < Last-Modified: Mon, 16 Jan 2023 16:42:46 GMT
2023-01-16T17:06:36.305Z [CURL DBG] < Content-Length: 1426
2023-01-16T17:06:36.305Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:36 GMT
2023-01-16T17:06:36.305Z [CURL DBG] < Server: EC2ws
2023-01-16T17:06:36.305Z [CURL DBG] < Connection: close
2023-01-16T17:06:36.305Z [CURL DBG] < 
2023-01-16T17:06:36.305Z [CURL DBG] * Closing connection 3
2023-01-16T17:06:36.305Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-16T17:06:36.305Z [INF]       s3fs_cred.cpp:SetIAMCredentials(550): IAM credential response = "{
  "Code" : "Success",
  "LastUpdated" : "2023-01-16T16:43:37Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "<<REDACTED-ACCESS-KEY>>",
  "SecretAccessKey" : "<<REDACTED-SECRETE-ACCESS-KEY>>",
  "Token" : "<<REDACTED-ROLE-TOKEN>>",
  "Expiration" : "2023-01-16T23:17:46Z"
}"
2023-01-16T17:06:36.305Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1166): IAM Access Token refreshed
2023-01-16T17:06:36.305Z [INF]       curl.cpp:CheckBucket(3519): check a bucket.
2023-01-16T17:06:36.305Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/
2023-01-16T17:06:36.305Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/
2023-01-16T17:06:36.305Z [INF]       curl.cpp:insertV4Headers(2741): computing signature [GET] [/] [] []
2023-01-16T17:06:36.305Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.amazonaws.com
2023-01-16T17:06:36.307Z [CURL DBG] *   Trying 52.219.108.249:443...
2023-01-16T17:06:36.307Z [CURL DBG] * Connected to s3.us-east-2.amazonaws.com (52.219.108.249) port 443 (#4)
2023-01-16T17:06:36.314Z [CURL DBG] * successfully set certificate verify locations:
2023-01-16T17:06:36.314Z [CURL DBG] *  CAfile: /etc/ssl/certs/ca-certificates.crt
2023-01-16T17:06:36.314Z [CURL DBG] *  CApath: /etc/ssl/certs
2023-01-16T17:06:36.314Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
2023-01-16T17:06:36.315Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2):
2023-01-16T17:06:36.315Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11):
2023-01-16T17:06:36.316Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
2023-01-16T17:06:36.316Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14):
2023-01-16T17:06:36.317Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
2023-01-16T17:06:36.317Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
2023-01-16T17:06:36.317Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20):
2023-01-16T17:06:36.318Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20):
2023-01-16T17:06:36.318Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
2023-01-16T17:06:36.318Z [CURL DBG] * Server certificate:
2023-01-16T17:06:36.318Z [CURL DBG] *  subject: CN=*.s3.us-east-2.amazonaws.com
2023-01-16T17:06:36.318Z [CURL DBG] *  start date: Sep 21 00:00:00 2022 GMT
2023-01-16T17:06:36.318Z [CURL DBG] *  expire date: Aug 31 23:59:59 2023 GMT
2023-01-16T17:06:36.318Z [CURL DBG] *  subjectAltName: host "s3.us-east-2.amazonaws.com" matched cert's "s3.us-east-2.amazonaws.com"
2023-01-16T17:06:36.318Z [CURL DBG] *  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
2023-01-16T17:06:36.318Z [CURL DBG] *  SSL certificate verify ok.
2023-01-16T17:06:36.318Z [CURL DBG] > GET /<<REDACTED-BUCKET-NAME>>/ HTTP/1.1
2023-01-16T17:06:36.318Z [CURL DBG] > Host: s3.us-east-2.amazonaws.com
2023-01-16T17:06:36.318Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-16T17:06:36.318Z [CURL DBG] > Accept: */*
2023-01-16T17:06:36.318Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<REDACTED-CREDENTIALS>>/20230116/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=<<REDACTED-SIGNATURE>>
2023-01-16T17:06:36.318Z [CURL DBG] > x-amz-content-sha256: <<REDACTED-SIGNATURE-SHA256>>
2023-01-16T17:06:36.318Z [CURL DBG] > x-amz-date: 20230116T170636Z
2023-01-16T17:06:36.318Z [CURL DBG] > x-amz-security-token: <<REDACTED-ROLE-TOKEN>>
2023-01-16T17:06:36.319Z [CURL DBG] > 
2023-01-16T17:06:36.322Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-16T17:06:36.322Z [CURL DBG] < HTTP/1.1 400 Bad Request
2023-01-16T17:06:36.322Z [CURL DBG] < x-amz-bucket-region: us-east-2
2023-01-16T17:06:36.322Z [CURL DBG] < x-amz-request-id: <<REDACTED-SIGNATURE-SHA256>>
2023-01-16T17:06:36.322Z [CURL DBG] < x-amz-id-2: <<REDACTED-ROLE-TOKEN>>
2023-01-16T17:06:36.322Z [CURL DBG] < Content-Type: application/xml
2023-01-16T17:06:36.322Z [CURL DBG] < Transfer-Encoding: chunked
2023-01-16T17:06:36.322Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:35 GMT
2023-01-16T17:06:36.322Z [CURL DBG] < Server: AmazonS3
2023-01-16T17:06:36.322Z [CURL DBG] < Connection: close
2023-01-16T17:06:36.322Z [CURL DBG] < 
2023-01-16T17:06:36.322Z [CURL DBG] * Closing connection 4
2023-01-16T17:06:36.322Z [CURL DBG] * TLSv1.2 (OUT), TLS alert, close notify (256):
2023-01-16T17:06:36.323Z [ERR] curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'</Message><Region>us-east-2</Region><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error>
2023-01-16T17:06:36.323Z [ERR] curl.cpp:CheckBucket(3566): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'</Message><Region>us-east-2</Region><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error>
2023-01-16T17:06:36.323Z [CRT] s3fs.cpp:s3fs_check_service(4093): Failed to connect region 'us-east-1'(default), so retry to connect region 'us-east-2'.
2023-01-16T17:06:36.323Z [INF]       curl.cpp:CheckBucket(3519): check a bucket.
2023-01-16T17:06:36.323Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/
2023-01-16T17:06:36.324Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/
2023-01-16T17:06:36.324Z [INF]       curl.cpp:insertV4Headers(2741): computing signature [GET] [/] [] []
2023-01-16T17:06:36.324Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.amazonaws.com
2023-01-16T17:06:36.324Z [CURL DBG] * Hostname s3.us-east-2.amazonaws.com was found in DNS cache
2023-01-16T17:06:36.324Z [CURL DBG] *   Trying 52.219.108.249:443...
2023-01-16T17:06:36.325Z [CURL DBG] * Connected to s3.us-east-2.amazonaws.com (52.219.108.249) port 443 (#5)
2023-01-16T17:06:36.331Z [CURL DBG] * successfully set certificate verify locations:
2023-01-16T17:06:36.331Z [CURL DBG] *  CAfile: /etc/ssl/certs/ca-certificates.crt
2023-01-16T17:06:36.331Z [CURL DBG] *  CApath: /etc/ssl/certs
2023-01-16T17:06:36.331Z [CURL DBG] * SSL re-using session ID
2023-01-16T17:06:36.331Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
2023-01-16T17:06:36.332Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2):
2023-01-16T17:06:36.332Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11):
2023-01-16T17:06:36.333Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
2023-01-16T17:06:36.333Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14):
2023-01-16T17:06:36.334Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
2023-01-16T17:06:36.334Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
2023-01-16T17:06:36.334Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20):
2023-01-16T17:06:36.335Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20):
2023-01-16T17:06:36.335Z [CURL DBG] * old SSL session ID is stale, removing
2023-01-16T17:06:36.335Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
2023-01-16T17:06:36.335Z [CURL DBG] * Server certificate:
2023-01-16T17:06:36.335Z [CURL DBG] *  subject: CN=*.s3.us-east-2.amazonaws.com
2023-01-16T17:06:36.335Z [CURL DBG] *  start date: Sep 21 00:00:00 2022 GMT
2023-01-16T17:06:36.335Z [CURL DBG] *  expire date: Aug 31 23:59:59 2023 GMT
2023-01-16T17:06:36.335Z [CURL DBG] *  subjectAltName: host "s3.us-east-2.amazonaws.com" matched cert's "s3.us-east-2.amazonaws.com"
2023-01-16T17:06:36.335Z [CURL DBG] *  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
2023-01-16T17:06:36.335Z [CURL DBG] *  SSL certificate verify ok.
2023-01-16T17:06:36.335Z [CURL DBG] > GET /<<REDACTED-BUCKET-NAME>>/ HTTP/1.1
2023-01-16T17:06:36.335Z [CURL DBG] > Host: s3.us-east-2.amazonaws.com
2023-01-16T17:06:36.335Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-16T17:06:36.335Z [CURL DBG] > Accept: */*
2023-01-16T17:06:36.335Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<REDACTED-CREDENTIALS>>/20230116/us-east-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=<<REDACTED-SIGNATURE>>
2023-01-16T17:06:36.335Z [CURL DBG] > x-amz-content-sha256: <<REDACTED-SIGNATURE-SHA256>>
2023-01-16T17:06:36.335Z [CURL DBG] > x-amz-date: 20230116T170636Z
2023-01-16T17:06:36.336Z [CURL DBG] > x-amz-security-token: <<REDACTED-ROLE-TOKEN>>
2023-01-16T17:06:36.336Z [CURL DBG] > 
2023-01-16T17:06:36.419Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-16T17:06:36.419Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-16T17:06:36.419Z [CURL DBG] < x-amz-id-2: <<REDACTED-AMZ-ID>>
2023-01-16T17:06:36.419Z [CURL DBG] < x-amz-request-id: <<REDACTED-REQUEST-ID>>
2023-01-16T17:06:36.419Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:37 GMT
2023-01-16T17:06:36.419Z [CURL DBG] < x-amz-bucket-region: us-east-2
2023-01-16T17:06:36.419Z [CURL DBG] < Content-Type: application/xml
2023-01-16T17:06:36.419Z [CURL DBG] < Transfer-Encoding: chunked
2023-01-16T17:06:36.419Z [CURL DBG] < Server: AmazonS3
2023-01-16T17:06:36.419Z [CURL DBG] < 
2023-01-16T17:06:36.424Z [CURL DBG] * Connection #5 to host s3.us-east-2.amazonaws.com left intact
2023-01-16T17:06:36.425Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-16T17:06:36.428Z [INF] s3fs.cpp:s3fs_destroy(3958): destroy

For the record, both instance and bucket are in us-east-2. If I do not specify the '-o url=https://s3.us-east-2.amazonaws.com/' option I get similar behaviour but after the two CTRL-C the log ends with:

2023-01-16T17:11:28.537Z [ERR] curl.cpp:RequestPerform(2396): HTTP response code 301(Moved Permanently: also happens when bucket's region is incorrect), returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint><<REDACTED-BUCKET-NAME>>.s3.amazonaws.com</Endpoint><Bucket><<REDACTED-BUCKET-NAME>></Bucket><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error>
2023-01-16T17:11:28.537Z [ERR] curl.cpp:RequestPerform(2397): The options of url and endpoint may be useful for solving, please try to use both options.
2023-01-16T17:11:28.537Z [ERR] curl.cpp:CheckBucket(3566): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint><<REDACTED-BUCKET-NAME>>.s3.amazonaws.com</Endpoint><Bucket><<REDACTED-BUCKET-NAME>></Bucket><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error>
2023-01-16T17:11:28.537Z [ERR] s3fs.cpp:s3fs_check_service(4107): S3 service returned PermanentRedirect with endpoint: <<REDACTED-BUCKET-NAME>>.s3.amazonaws.com

aws cli is working as expected and can see the entire bucket content:

$ aws s3 ls <<REDACTED-BUCKET-NAME>>
                           PRE <<REDACTED-BUCKET-CONTENT>>/
<!-- gh-comment-id:1384365956 --> @porelli commented on GitHub (Jan 16, 2023): Thank you for checking! If it was, I'm not sure why the Debian instance had IMDSv2 disabled. Surely it should have had at least the IMDSv1 enabled but I do not have the instance anymore to re-try. You are 100% correct, with the "-o imdsv1only" I can successfully mount the bucket. Altough, I'm not able to see the entire content. I suspect this might be because it was created manuall and not using using s3fs. Regarding IMDSv2, I collected some more data that might be helpful for further investigation. I just tried again with the latest code in the git repo and I can still reproduce the same problem on a new instance that definitely has both IMDSv1 and IMDSv2 enabled: IMDSv1: ``` $ curl http://169.254.169.254/latest/meta-data/ ami-id ami-launch-index ami-manifest-path block-device-mapping/ events/ hostname iam/ identity-credentials/ instance-action instance-id instance-life-cycle instance-type local-hostname local-ipv4 mac metrics/ network/ placement/ profile public-hostname public-ipv4 public-keys/ reservation-id security-groups services/ system ``` IMDSv2: ``` ~/s3fs-fuse/src$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/ % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 56 100 56 0 0 14000 0 --:--:-- --:--:-- --:--:-- 14000 * Trying 169.254.169.254:80... * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) > GET /latest/meta-data/ HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.74.0 > Accept: */* > X-aws-ec2-metadata-token: <<REDACTED-TOKEN>> > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 < Content-Type: text/plain < Accept-Ranges: none < Last-Modified: Mon, 16 Jan 2023 16:42:46 GMT < Content-Length: 325 < Date: Mon, 16 Jan 2023 16:57:34 GMT < Server: EC2ws < Connection: close < ami-id ami-launch-index ami-manifest-path block-device-mapping/ events/ hostname iam/ identity-credentials/ instance-action instance-id instance-life-cycle instance-type local-hostname local-ipv4 mac metrics/ network/ placement/ profile public-hostname public-ipv4 public-keys/ reservation-id security-groups services/ * Closing connection 0 system ``` IMDSv2 with wrong token: ``` $ TOKEN=fake && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/ * Trying 169.254.169.254:80... * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) > GET /latest/meta-data/ HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.74.0 > Accept: */* > X-aws-ec2-metadata-token: fake > * Mark bundle as not supporting multiuse < HTTP/1.1 401 Unauthorized < Content-Length: 0 < Date: Mon, 16 Jan 2023 17:00:51 GMT < Server: EC2ws < Connection: close < Content-Type: text/plain < * Closing connection 0 ``` S3FS hangs as @jpedrick reported in this ticket: ``` $ sudo ./s3fs <<REDACTED-BUCKET-NAME>> /tmp/bucket/ -o iam_role=auto -d -f -s -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.us-east-2.amazonaws.com/ 2023-01-16T17:03:40.218Z [INF] s3fs version 1.91(16bc449) : s3fs -o iam_role=auto -d -f -s -o allow_other -o use_path_request_style -o umask=022 -o curldbg -o dbglevel=info -o url=https://s3.us-east-2.amazonaws.com/ <<REDACTED-BUCKET-NAME>> /tmp/bucket/ 2023-01-16T17:03:40.219Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [INF] 2023-01-16T17:03:40.219Z [INF] s3fs.cpp:set_mountpoint_attribute(4191): PROC(uid=0, gid=0) - MountPoint(uid=1000, gid=1000, mode=40755) 2023-01-16T17:03:40.221Z [INF] curl.cpp:InitMimeType(428): Loaded mime information from /etc/mime.types 2023-01-16T17:03:40.222Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(78): The path to cache top dir is empty, thus not need to check permission. 2023-01-16T17:03:40.222Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do. 2023-01-16T17:03:40.224Z [CRT] s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function. 2023-01-16T17:03:40.224Z [INF] s3fs.cpp:s3fs_init(3914): init v1.91(commit:16bc449) with OpenSSL, credential-library(built-in) 2023-01-16T17:03:40.224Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-16T17:03:40.225Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2023-01-16T17:03:40.225Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2023-01-16T17:03:40.225Z [CURL DBG] > Host: 169.254.169.254 2023-01-16T17:03:40.225Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-16T17:03:40.225Z [CURL DBG] > Accept: */* 2023-01-16T17:03:40.225Z [CURL DBG] > Transfer-Encoding: chunked 2023-01-16T17:03:40.225Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2023-01-16T17:03:40.225Z [CURL DBG] > 2023-01-16T17:03:40.225Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-16T17:03:40.225Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-16T17:03:40.225Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-16T17:03:40.225Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-16T17:03:40.225Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. ``` If I press ctrl+c it continues but then hangs again: ``` ^C2023-01-16T17:05:07.680Z [CURL DBG] * Signaling end of chunked upload via terminating chunk. 2023-01-16T17:05:07.681Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-16T17:05:07.681Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-16T17:05:07.681Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-01-16T17:05:07.681Z [CURL DBG] < Content-Length: 56 2023-01-16T17:05:07.681Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:05:07 GMT 2023-01-16T17:05:07.681Z [CURL DBG] < Server: EC2ws 2023-01-16T17:05:07.681Z [CURL DBG] < Connection: close 2023-01-16T17:05:07.681Z [CURL DBG] < Content-Type: text/plain 2023-01-16T17:05:07.681Z [CURL DBG] < 2023-01-16T17:05:07.681Z [CURL DBG] * Closing connection 0 2023-01-16T17:05:07.681Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-16T17:05:07.681Z [INF] s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to <<REDACTED-TOKEN>> 2023-01-16T17:05:07.681Z [INF] curl.cpp:GetIAMRoleFromMetaData(3006): Get IAM Role name 2023-01-16T17:05:07.681Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-16T17:05:07.681Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1) 2023-01-16T17:05:07.681Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1 2023-01-16T17:05:07.681Z [CURL DBG] > Host: 169.254.169.254 2023-01-16T17:05:07.681Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-16T17:05:07.681Z [CURL DBG] > Accept: */* 2023-01-16T17:05:07.681Z [CURL DBG] > X-aws-ec2-metadata-token: <<REDACTED-TOKEN>> 2023-01-16T17:05:07.681Z [CURL DBG] > 2023-01-16T17:05:07.682Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-16T17:05:07.682Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-16T17:05:07.682Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-01-16T17:05:07.682Z [CURL DBG] < Content-Type: text/plain 2023-01-16T17:05:07.682Z [CURL DBG] < Accept-Ranges: none 2023-01-16T17:05:07.682Z [CURL DBG] < Last-Modified: Mon, 16 Jan 2023 16:42:46 GMT 2023-01-16T17:05:07.682Z [CURL DBG] < Content-Length: 19 2023-01-16T17:05:07.682Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:05:07 GMT 2023-01-16T17:05:07.682Z [CURL DBG] < Server: EC2ws 2023-01-16T17:05:07.682Z [CURL DBG] < Connection: close 2023-01-16T17:05:07.682Z [CURL DBG] < 2023-01-16T17:05:07.682Z [CURL DBG] * Closing connection 1 2023-01-16T17:05:07.682Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-16T17:05:07.682Z [INF] s3fs_cred.cpp:SetIAMRoleFromMetaData(582): IAM role name response = "<<REDACTED-BUCKET-NAME>>" 2023-01-16T17:05:07.682Z [INF] s3fs_cred.cpp:LoadIAMRoleFromMetaData(543): loaded IAM role name = <<REDACTED-BUCKET-NAME>> 2023-01-16T17:05:07.682Z [INF] s3fs.cpp:s3fs_check_service(4058): check services. 2023-01-16T17:05:07.682Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1152): IAM Access Token refreshing... 2023-01-16T17:05:07.682Z [INF] s3fs_cred.cpp:GetIAMCredentialsURL(374): [IAM role=<<REDACTED-BUCKET-NAME>>] 2023-01-16T17:05:07.682Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache 2023-01-16T17:05:07.682Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-16T17:05:07.682Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#2) 2023-01-16T17:05:07.682Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2023-01-16T17:05:07.682Z [CURL DBG] > Host: 169.254.169.254 2023-01-16T17:05:07.682Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-16T17:05:07.682Z [CURL DBG] > Accept: */* 2023-01-16T17:05:07.682Z [CURL DBG] > Transfer-Encoding: chunked 2023-01-16T17:05:07.682Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2023-01-16T17:05:07.682Z [CURL DBG] > ``` At this point, if I try to access the folder from another terminal, the shell hangs there. Pressing CTRL+C once more in the original shell: ``` ^C2023-01-16T17:06:36.304Z [CURL DBG] * Signaling end of chunked upload via terminating chunk. 2023-01-16T17:06:36.304Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-16T17:06:36.304Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-16T17:06:36.304Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-01-16T17:06:36.305Z [CURL DBG] < Content-Length: 56 2023-01-16T17:06:36.305Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:36 GMT 2023-01-16T17:06:36.305Z [CURL DBG] < Server: EC2ws 2023-01-16T17:06:36.305Z [CURL DBG] < Connection: close 2023-01-16T17:06:36.305Z [CURL DBG] < Content-Type: text/plain 2023-01-16T17:06:36.305Z [CURL DBG] < 2023-01-16T17:06:36.305Z [CURL DBG] * Closing connection 2 2023-01-16T17:06:36.305Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-16T17:06:36.305Z [INF] s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to <<REDACTED-TOKEN>> 2023-01-16T17:06:36.305Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-16T17:06:36.305Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#3) 2023-01-16T17:06:36.305Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/<<REDACTED-BUCKET-NAME>> HTTP/1.1 2023-01-16T17:06:36.305Z [CURL DBG] > Host: 169.254.169.254 2023-01-16T17:06:36.305Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-16T17:06:36.305Z [CURL DBG] > Accept: */* 2023-01-16T17:06:36.305Z [CURL DBG] > X-aws-ec2-metadata-token: <<REDACTED-TOKEN>> 2023-01-16T17:06:36.305Z [CURL DBG] > 2023-01-16T17:06:36.305Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-16T17:06:36.305Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-16T17:06:36.305Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-01-16T17:06:36.305Z [CURL DBG] < Content-Type: text/plain 2023-01-16T17:06:36.305Z [CURL DBG] < Accept-Ranges: none 2023-01-16T17:06:36.305Z [CURL DBG] < Last-Modified: Mon, 16 Jan 2023 16:42:46 GMT 2023-01-16T17:06:36.305Z [CURL DBG] < Content-Length: 1426 2023-01-16T17:06:36.305Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:36 GMT 2023-01-16T17:06:36.305Z [CURL DBG] < Server: EC2ws 2023-01-16T17:06:36.305Z [CURL DBG] < Connection: close 2023-01-16T17:06:36.305Z [CURL DBG] < 2023-01-16T17:06:36.305Z [CURL DBG] * Closing connection 3 2023-01-16T17:06:36.305Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-16T17:06:36.305Z [INF] s3fs_cred.cpp:SetIAMCredentials(550): IAM credential response = "{ "Code" : "Success", "LastUpdated" : "2023-01-16T16:43:37Z", "Type" : "AWS-HMAC", "AccessKeyId" : "<<REDACTED-ACCESS-KEY>>", "SecretAccessKey" : "<<REDACTED-SECRETE-ACCESS-KEY>>", "Token" : "<<REDACTED-ROLE-TOKEN>>", "Expiration" : "2023-01-16T23:17:46Z" }" 2023-01-16T17:06:36.305Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1166): IAM Access Token refreshed 2023-01-16T17:06:36.305Z [INF] curl.cpp:CheckBucket(3519): check a bucket. 2023-01-16T17:06:36.305Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/ 2023-01-16T17:06:36.305Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/ 2023-01-16T17:06:36.305Z [INF] curl.cpp:insertV4Headers(2741): computing signature [GET] [/] [] [] 2023-01-16T17:06:36.305Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.amazonaws.com 2023-01-16T17:06:36.307Z [CURL DBG] * Trying 52.219.108.249:443... 2023-01-16T17:06:36.307Z [CURL DBG] * Connected to s3.us-east-2.amazonaws.com (52.219.108.249) port 443 (#4) 2023-01-16T17:06:36.314Z [CURL DBG] * successfully set certificate verify locations: 2023-01-16T17:06:36.314Z [CURL DBG] * CAfile: /etc/ssl/certs/ca-certificates.crt 2023-01-16T17:06:36.314Z [CURL DBG] * CApath: /etc/ssl/certs 2023-01-16T17:06:36.314Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1): 2023-01-16T17:06:36.315Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2): 2023-01-16T17:06:36.315Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11): 2023-01-16T17:06:36.316Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): 2023-01-16T17:06:36.316Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14): 2023-01-16T17:06:36.317Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): 2023-01-16T17:06:36.317Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): 2023-01-16T17:06:36.317Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20): 2023-01-16T17:06:36.318Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20): 2023-01-16T17:06:36.318Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 2023-01-16T17:06:36.318Z [CURL DBG] * Server certificate: 2023-01-16T17:06:36.318Z [CURL DBG] * subject: CN=*.s3.us-east-2.amazonaws.com 2023-01-16T17:06:36.318Z [CURL DBG] * start date: Sep 21 00:00:00 2022 GMT 2023-01-16T17:06:36.318Z [CURL DBG] * expire date: Aug 31 23:59:59 2023 GMT 2023-01-16T17:06:36.318Z [CURL DBG] * subjectAltName: host "s3.us-east-2.amazonaws.com" matched cert's "s3.us-east-2.amazonaws.com" 2023-01-16T17:06:36.318Z [CURL DBG] * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon 2023-01-16T17:06:36.318Z [CURL DBG] * SSL certificate verify ok. 2023-01-16T17:06:36.318Z [CURL DBG] > GET /<<REDACTED-BUCKET-NAME>>/ HTTP/1.1 2023-01-16T17:06:36.318Z [CURL DBG] > Host: s3.us-east-2.amazonaws.com 2023-01-16T17:06:36.318Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-16T17:06:36.318Z [CURL DBG] > Accept: */* 2023-01-16T17:06:36.318Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<REDACTED-CREDENTIALS>>/20230116/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=<<REDACTED-SIGNATURE>> 2023-01-16T17:06:36.318Z [CURL DBG] > x-amz-content-sha256: <<REDACTED-SIGNATURE-SHA256>> 2023-01-16T17:06:36.318Z [CURL DBG] > x-amz-date: 20230116T170636Z 2023-01-16T17:06:36.318Z [CURL DBG] > x-amz-security-token: <<REDACTED-ROLE-TOKEN>> 2023-01-16T17:06:36.319Z [CURL DBG] > 2023-01-16T17:06:36.322Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-16T17:06:36.322Z [CURL DBG] < HTTP/1.1 400 Bad Request 2023-01-16T17:06:36.322Z [CURL DBG] < x-amz-bucket-region: us-east-2 2023-01-16T17:06:36.322Z [CURL DBG] < x-amz-request-id: <<REDACTED-SIGNATURE-SHA256>> 2023-01-16T17:06:36.322Z [CURL DBG] < x-amz-id-2: <<REDACTED-ROLE-TOKEN>> 2023-01-16T17:06:36.322Z [CURL DBG] < Content-Type: application/xml 2023-01-16T17:06:36.322Z [CURL DBG] < Transfer-Encoding: chunked 2023-01-16T17:06:36.322Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:35 GMT 2023-01-16T17:06:36.322Z [CURL DBG] < Server: AmazonS3 2023-01-16T17:06:36.322Z [CURL DBG] < Connection: close 2023-01-16T17:06:36.322Z [CURL DBG] < 2023-01-16T17:06:36.322Z [CURL DBG] * Closing connection 4 2023-01-16T17:06:36.322Z [CURL DBG] * TLSv1.2 (OUT), TLS alert, close notify (256): 2023-01-16T17:06:36.323Z [ERR] curl.cpp:RequestPerform(2410): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'</Message><Region>us-east-2</Region><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error> 2023-01-16T17:06:36.323Z [ERR] curl.cpp:CheckBucket(3566): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-east-2'</Message><Region>us-east-2</Region><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error> 2023-01-16T17:06:36.323Z [CRT] s3fs.cpp:s3fs_check_service(4093): Failed to connect region 'us-east-1'(default), so retry to connect region 'us-east-2'. 2023-01-16T17:06:36.323Z [INF] curl.cpp:CheckBucket(3519): check a bucket. 2023-01-16T17:06:36.323Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/ 2023-01-16T17:06:36.324Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://s3.us-east-2.amazonaws.com/<<REDACTED-BUCKET-NAME>>/ 2023-01-16T17:06:36.324Z [INF] curl.cpp:insertV4Headers(2741): computing signature [GET] [/] [] [] 2023-01-16T17:06:36.324Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.amazonaws.com 2023-01-16T17:06:36.324Z [CURL DBG] * Hostname s3.us-east-2.amazonaws.com was found in DNS cache 2023-01-16T17:06:36.324Z [CURL DBG] * Trying 52.219.108.249:443... 2023-01-16T17:06:36.325Z [CURL DBG] * Connected to s3.us-east-2.amazonaws.com (52.219.108.249) port 443 (#5) 2023-01-16T17:06:36.331Z [CURL DBG] * successfully set certificate verify locations: 2023-01-16T17:06:36.331Z [CURL DBG] * CAfile: /etc/ssl/certs/ca-certificates.crt 2023-01-16T17:06:36.331Z [CURL DBG] * CApath: /etc/ssl/certs 2023-01-16T17:06:36.331Z [CURL DBG] * SSL re-using session ID 2023-01-16T17:06:36.331Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1): 2023-01-16T17:06:36.332Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2): 2023-01-16T17:06:36.332Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11): 2023-01-16T17:06:36.333Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): 2023-01-16T17:06:36.333Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14): 2023-01-16T17:06:36.334Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): 2023-01-16T17:06:36.334Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): 2023-01-16T17:06:36.334Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20): 2023-01-16T17:06:36.335Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20): 2023-01-16T17:06:36.335Z [CURL DBG] * old SSL session ID is stale, removing 2023-01-16T17:06:36.335Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 2023-01-16T17:06:36.335Z [CURL DBG] * Server certificate: 2023-01-16T17:06:36.335Z [CURL DBG] * subject: CN=*.s3.us-east-2.amazonaws.com 2023-01-16T17:06:36.335Z [CURL DBG] * start date: Sep 21 00:00:00 2022 GMT 2023-01-16T17:06:36.335Z [CURL DBG] * expire date: Aug 31 23:59:59 2023 GMT 2023-01-16T17:06:36.335Z [CURL DBG] * subjectAltName: host "s3.us-east-2.amazonaws.com" matched cert's "s3.us-east-2.amazonaws.com" 2023-01-16T17:06:36.335Z [CURL DBG] * issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon 2023-01-16T17:06:36.335Z [CURL DBG] * SSL certificate verify ok. 2023-01-16T17:06:36.335Z [CURL DBG] > GET /<<REDACTED-BUCKET-NAME>>/ HTTP/1.1 2023-01-16T17:06:36.335Z [CURL DBG] > Host: s3.us-east-2.amazonaws.com 2023-01-16T17:06:36.335Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-16T17:06:36.335Z [CURL DBG] > Accept: */* 2023-01-16T17:06:36.335Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<REDACTED-CREDENTIALS>>/20230116/us-east-2/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=<<REDACTED-SIGNATURE>> 2023-01-16T17:06:36.335Z [CURL DBG] > x-amz-content-sha256: <<REDACTED-SIGNATURE-SHA256>> 2023-01-16T17:06:36.335Z [CURL DBG] > x-amz-date: 20230116T170636Z 2023-01-16T17:06:36.336Z [CURL DBG] > x-amz-security-token: <<REDACTED-ROLE-TOKEN>> 2023-01-16T17:06:36.336Z [CURL DBG] > 2023-01-16T17:06:36.419Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-16T17:06:36.419Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-16T17:06:36.419Z [CURL DBG] < x-amz-id-2: <<REDACTED-AMZ-ID>> 2023-01-16T17:06:36.419Z [CURL DBG] < x-amz-request-id: <<REDACTED-REQUEST-ID>> 2023-01-16T17:06:36.419Z [CURL DBG] < Date: Mon, 16 Jan 2023 17:06:37 GMT 2023-01-16T17:06:36.419Z [CURL DBG] < x-amz-bucket-region: us-east-2 2023-01-16T17:06:36.419Z [CURL DBG] < Content-Type: application/xml 2023-01-16T17:06:36.419Z [CURL DBG] < Transfer-Encoding: chunked 2023-01-16T17:06:36.419Z [CURL DBG] < Server: AmazonS3 2023-01-16T17:06:36.419Z [CURL DBG] < 2023-01-16T17:06:36.424Z [CURL DBG] * Connection #5 to host s3.us-east-2.amazonaws.com left intact 2023-01-16T17:06:36.425Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-16T17:06:36.428Z [INF] s3fs.cpp:s3fs_destroy(3958): destroy ``` For the record, both instance and bucket are in us-east-2. If I do not specify the '-o url=https://s3.us-east-2.amazonaws.com/' option I get similar behaviour but after the two CTRL-C the log ends with: ``` 2023-01-16T17:11:28.537Z [ERR] curl.cpp:RequestPerform(2396): HTTP response code 301(Moved Permanently: also happens when bucket's region is incorrect), returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint><<REDACTED-BUCKET-NAME>>.s3.amazonaws.com</Endpoint><Bucket><<REDACTED-BUCKET-NAME>></Bucket><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error> 2023-01-16T17:11:28.537Z [ERR] curl.cpp:RequestPerform(2397): The options of url and endpoint may be useful for solving, please try to use both options. 2023-01-16T17:11:28.537Z [ERR] curl.cpp:CheckBucket(3566): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Endpoint><<REDACTED-BUCKET-NAME>>.s3.amazonaws.com</Endpoint><Bucket><<REDACTED-BUCKET-NAME>></Bucket><RequestId><<REDACTED-REQUEST-ID>></RequestId><HostId><<REDACTED-HOST-ID>></HostId></Error> 2023-01-16T17:11:28.537Z [ERR] s3fs.cpp:s3fs_check_service(4107): S3 service returned PermanentRedirect with endpoint: <<REDACTED-BUCKET-NAME>>.s3.amazonaws.com ``` aws cli is working as expected and can see the entire bucket content: ``` $ aws s3 ls <<REDACTED-BUCKET-NAME>> PRE <<REDACTED-BUCKET-CONTENT>>/ ```
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 22, 2023):

You are 100% correct, with the "-o imdsv1only" I can successfully mount the bucket. Altough, I'm not able to see the entire content. I suspect this might be because it was created manuall and not using using s3fs.

I think that the not-found objects created using tools other than s3fs can be resolved by adding the compat_dir option.
(Because the default behavior of s3fs has changed, objects will be invisible if this option is not given.)

IMDSv2 with wrong token:

We will continue to investigate this report.

<!-- gh-comment-id:1399428803 --> @ggtakec commented on GitHub (Jan 22, 2023): > You are 100% correct, with the "-o imdsv1only" I can successfully mount the bucket. Altough, I'm not able to see the entire content. I suspect this might be because it was created manuall and not using using s3fs. I think that the not-found objects created using tools other than s3fs can be resolved by adding the `compat_dir` option. _(Because the default behavior of s3fs has changed, objects will be invisible if this option is not given.)_ > IMDSv2 with wrong token: We will continue to investigate this report.
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 12, 2023):

I believe this issue is resolved by merging #2103. (See #2098)
This issue will be closed, but please reopen it if you still have problems.

<!-- gh-comment-id:1426933602 --> @ggtakec commented on GitHub (Feb 12, 2023): I believe this issue is resolved by merging #2103. (See #2098) This issue will be closed, but please reopen it if you still have problems.
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@jacky96623 commented on GitHub (Feb 13, 2023):

Hi @ggtakec , thanks for the fix! May I know usually it takes how much time for the latest change to be reflected in Linux repository?

<!-- gh-comment-id:1427209058 --> @jacky96623 commented on GitHub (Feb 13, 2023): Hi @ggtakec , thanks for the fix! May I know usually it takes how much time for the latest change to be reflected in Linux repository?
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 19, 2023):

@jacky96623
I think it will be reflected after we release(tag)...It is a few weeks?
(Official packaging is done by another great collaborator, so I can't state in detail)

<!-- gh-comment-id:1435910963 --> @ggtakec commented on GitHub (Feb 19, 2023): @jacky96623 I think it will be reflected after we release(tag)...It is a few weeks? (Official packaging is done by another great collaborator, so I can't state in detail)
kerem commented 2026-03-04 01:50:43 +03:00
Author
Owner
Copy link

@jacky96623 commented on GitHub (Feb 19, 2023):

@ggtakec Thanks for the information. I asked the question since it appears to me that the latest version of s3fs (1.91), which was released nearly a year ago, is not yet available in the official repositories of some Linux, e.g., Ubuntu 22.04. I was just wondering if I should try building the latest version from source code by myself.

Perhaps I should just wait for some time more - hopefully it is just the repository owner/maintainer needing time to review/evaluate.

<!-- gh-comment-id:1435979788 --> @jacky96623 commented on GitHub (Feb 19, 2023): @ggtakec Thanks for the information. I asked the question since it appears to me that the latest version of s3fs (1.91), which was released nearly a year ago, is not yet available in the official repositories of some Linux, e.g., Ubuntu 22.04. I was just wondering if I should try building the latest version from source code by myself. Perhaps I should just wait for some time more - hopefully it is just the repository owner/maintainer needing time to review/evaluate.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1018
No description provided.