starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 05:16:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2098] PUT /latest/api/token hangs without -o imdsv1only #1068

New issue
Closed
opened 2026-03-04 01:51:06 +03:00 by kerem · 2 comments
kerem commented 2026-03-04 01:51:06 +03:00
Owner
Copy link

Originally created by @daverysyn on GitHub (Jan 25, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2098

Additional Information

Version of s3fs being used (s3fs --version)

I built s3fs from source but the behavior in the yum 1.91-1 package was identical.

$ s3fs --version
Amazon Simple Storage Service File System V1.91 (commit:16bc449) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

$ rpm -qi fuse
Name        : fuse
Version     : 2.9.2
Release     : 11.amzn2
Architecture: x86_64
Install Date: Wed Oct  5 08:55:26 2022
Group       : System Environment/Base
Size        : 222809
License     : GPL+
Signature   : RSA/SHA256, Thu Dec  6 19:31:53 2018, Key ID 11cf1f95c87f5b1a
Source RPM  : fuse-2.9.2-11.amzn2.src.rpm
Build Date  : Fri Nov 16 20:35:39 2018
Build Host  : build.amazon.com
Relocations : (not relocatable)
Packager    : Amazon Linux
Vendor      : Amazon Linux
URL         : https://github.com/libfuse/libfuse
Summary     : File System in Userspace (FUSE) utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE userspace tools to
mount a FUSE filesystem.

Kernel information (uname -r)

$ uname -r
5.10.102-99.473.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Curl info

$ yum list curl libcurl
Installed Packages
curl.x86_64                                                                  7.79.1-7.amzn2.0.1                                                                @amzn2-core
libcurl.x86_64                                                               7.79.1-7.amzn2.0.1                                                                @amzn2-core

How to run s3fs, if applicable

$ s3fs <bucket_name> <mount> -o use_path_request_style -o dbglevel=debug -o curldbg -f -s -o iam_role=auto -o ro

log messages (with my comments)

$ s3fs <bucket_name> <mount> -o use_path_request_style -o dbglevel=debug -o curldbg -f -s -o iam_role=auto -o ro
2023-01-24T19:13:38.073Z [INF] s3fs version 1.91(16bc449) : s3fs -o use_path_request_style -o dbglevel=debug -o curldbg -f -s -o iam_role=auto -o ro <bucket_name> <mount>
2023-01-24T19:13:38.075Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [DBG]
2023-01-24T19:13:38.075Z [INF]     s3fs.cpp:set_mountpoint_attribute(4192): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775)
2023-01-24T19:13:38.076Z [DBG] curl.cpp:InitMimeType(404): Try to load mime types from /etc/mime.types file.
2023-01-24T19:13:38.076Z [DBG] curl.cpp:InitMimeType(408): The old mime types are cleared to load new mime types.
2023-01-24T19:13:38.077Z [INF] curl.cpp:InitMimeType(428): Loaded mime information from /etc/mime.types
2023-01-24T19:13:38.077Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(78): The path to cache top dir is empty, thus not need to check permission.
2023-01-24T19:13:38.077Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do.
2023-01-24T19:13:38.078Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-24T19:13:38.078Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-24T19:13:38.078Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-24T19:13:38.078Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-24T19:13:38.078Z [INF]       threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan.
2023-01-24T19:13:38.081Z [CRT] s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function.
2023-01-24T19:13:38.081Z [INF] s3fs.cpp:s3fs_init(3914): init v1.91(commit:16bc449) with OpenSSL, credential-library(built-in)
2023-01-24T19:13:38.081Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-01-24T19:13:38.081Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/api/token
2023-01-24T19:13:38.081Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-24T19:13:38.082Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2023-01-24T19:13:38.082Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2023-01-24T19:13:38.082Z [CURL DBG] > Host: 169.254.169.254
2023-01-24T19:13:38.082Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-24T19:13:38.082Z [CURL DBG] > Accept: */*
2023-01-24T19:13:38.082Z [CURL DBG] > Transfer-Encoding: chunked
2023-01-24T19:13:38.082Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2023-01-24T19:13:38.082Z [CURL DBG] >
# Output hangs here and doesn't proceed.
# Add visual spacing by pressing enter a few times (doesn't alter behavior, as far as I can tell)


# CTRL+C twice seems to move things along (Pressing CTRL+C once only works if I don't use Enter to create visual spacing)
# It looks to me like this got back a 200, so why did the request hang?
^C^C2023-01-24T19:13:44.819Z [CURL DBG] * Signaling end of chunked upload via terminating chunk.
2023-01-24T19:13:44.819Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-24T19:13:44.823Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-24T19:13:44.823Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:40 GMT
2023-01-24T19:13:44.823Z [CURL DBG] < Server: EC2ws
2023-01-24T19:13:44.823Z [CURL DBG] < Connection: close
2023-01-24T19:13:44.823Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-01-24T19:13:44.823Z [CURL DBG] < Content-Type: text/plain
2023-01-24T19:13:44.823Z [CURL DBG] < Content-Length: 56
2023-01-24T19:13:44.823Z [CURL DBG] <
2023-01-24T19:13:44.823Z [CURL DBG] * Closing connection 0
2023-01-24T19:13:44.823Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-24T19:13:44.823Z [INF]       s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to REDACTED
2023-01-24T19:13:44.823Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-01-24T19:13:44.823Z [INF]       curl.cpp:GetIAMRoleFromMetaData(3006): Get IAM Role name
2023-01-24T19:13:44.823Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-01-24T19:13:44.823Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2023-01-24T19:13:44.823Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2023-01-24T19:13:44.823Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-24T19:13:44.823Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1)
2023-01-24T19:13:44.823Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
2023-01-24T19:13:44.823Z [CURL DBG] > Host: 169.254.169.254
2023-01-24T19:13:44.823Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-24T19:13:44.823Z [CURL DBG] > Accept: */*
2023-01-24T19:13:44.823Z [CURL DBG] > X-aws-ec2-metadata-token: REDACTED
2023-01-24T19:13:44.823Z [CURL DBG] >
2023-01-24T19:13:44.825Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-24T19:13:44.825Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-24T19:13:44.825Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:44 GMT
2023-01-24T19:13:44.825Z [CURL DBG] < Accept-Ranges: none
2023-01-24T19:13:44.825Z [CURL DBG] < Server: EC2ws
2023-01-24T19:13:44.825Z [CURL DBG] < Connection: close
2023-01-24T19:13:44.825Z [CURL DBG] < Last-Modified: Tue, 24 Jan 2023 18:45:22 GMT
2023-01-24T19:13:44.825Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21596
2023-01-24T19:13:44.825Z [CURL DBG] < Content-Type: text/plain
2023-01-24T19:13:44.825Z [CURL DBG] < Content-Length: 35
2023-01-24T19:13:44.825Z [CURL DBG] <
2023-01-24T19:13:44.825Z [CURL DBG] * Closing connection 1
2023-01-24T19:13:44.825Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-24T19:13:44.825Z [INF]       s3fs_cred.cpp:SetIAMRoleFromMetaData(582): IAM role name response = "<role>"
2023-01-24T19:13:44.825Z [INF] s3fs_cred.cpp:LoadIAMRoleFromMetaData(543): loaded IAM role name = <role>
2023-01-24T19:13:44.825Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-01-24T19:13:44.825Z [INF] s3fs.cpp:s3fs_check_service(4058): check services.
2023-01-24T19:13:44.825Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1152): IAM Access Token refreshing...
2023-01-24T19:13:44.825Z [INF]       s3fs_cred.cpp:GetIAMCredentialsURL(374): [IAM role=<role>]
2023-01-24T19:13:44.825Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-01-24T19:13:44.825Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/api/token
2023-01-24T19:13:44.825Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2023-01-24T19:13:44.825Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-24T19:13:44.825Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#2)
2023-01-24T19:13:44.825Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2023-01-24T19:13:44.825Z [CURL DBG] > Host: 169.254.169.254
2023-01-24T19:13:44.825Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-24T19:13:44.825Z [CURL DBG] > Accept: */*
2023-01-24T19:13:44.825Z [CURL DBG] > Transfer-Encoding: chunked
2023-01-24T19:13:44.825Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2023-01-24T19:13:44.825Z [CURL DBG] >
# This request also hangs
# Press enter a few times to create space



# CTRL+C twice to get things moving again.  Again, we see that a 200 came back on this request, so why did it hang up?
^C^C2023-01-24T19:13:55.386Z [CURL DBG] * Signaling end of chunked upload via terminating chunk.
2023-01-24T19:13:55.386Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-24T19:13:55.387Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-24T19:13:55.387Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:54 GMT
2023-01-24T19:13:55.387Z [CURL DBG] < Server: EC2ws
2023-01-24T19:13:55.387Z [CURL DBG] < Connection: close
2023-01-24T19:13:55.387Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
2023-01-24T19:13:55.387Z [CURL DBG] < Content-Type: text/plain
2023-01-24T19:13:55.387Z [CURL DBG] < Content-Length: 56
2023-01-24T19:13:55.387Z [CURL DBG] <
2023-01-24T19:13:55.387Z [CURL DBG] * Closing connection 2
2023-01-24T19:13:55.387Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-24T19:13:55.387Z [INF]       s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to REDACTED
2023-01-24T19:13:55.387Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-01-24T19:13:55.387Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-01-24T19:13:55.387Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/<role>
2023-01-24T19:13:55.387Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2023-01-24T19:13:55.387Z [CURL DBG] *   Trying 169.254.169.254:80...
2023-01-24T19:13:55.388Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#3)
2023-01-24T19:13:55.388Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/<role> HTTP/1.1
2023-01-24T19:13:55.388Z [CURL DBG] > Host: 169.254.169.254
2023-01-24T19:13:55.388Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-24T19:13:55.388Z [CURL DBG] > Accept: */*
2023-01-24T19:13:55.388Z [CURL DBG] > X-aws-ec2-metadata-token: REDACTED
2023-01-24T19:13:55.388Z [CURL DBG] >
2023-01-24T19:13:55.390Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-24T19:13:55.390Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-24T19:13:55.390Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:55 GMT
2023-01-24T19:13:55.390Z [CURL DBG] < Content-Type: text/plain
2023-01-24T19:13:55.390Z [CURL DBG] < Content-Length: 958
2023-01-24T19:13:55.390Z [CURL DBG] < Server: Jetty(9.4.z-SNAPSHOT)
2023-01-24T19:13:55.390Z [CURL DBG] <
2023-01-24T19:13:55.390Z [CURL DBG] * Connection #3 to host 169.254.169.254 left intact
2023-01-24T19:13:55.390Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-24T19:13:55.390Z [INF]       s3fs_cred.cpp:SetIAMCredentials(550): IAM credential response = "{"AccessKeyId":"REDACTED","SecretAccessKey":"REDACTED","Token":"REDACTED","Expiration":"2023-01-24T20:13:30Z","Code":"Success","Message":null}"
2023-01-24T19:13:55.390Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-01-24T19:13:55.390Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1166): IAM Access Token refreshed
2023-01-24T19:13:55.390Z [INF]       curl.cpp:CheckBucket(3519): check a bucket.
2023-01-24T19:13:55.390Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31
2023-01-24T19:13:55.390Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/<bucket_name>/
2023-01-24T19:13:55.390Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://s3.amazonaws.com/<bucket_name>/
2023-01-24T19:13:55.390Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL https://s3.amazonaws.com/<bucket_name>/
2023-01-24T19:13:55.390Z [INF]       curl.cpp:insertV4Headers(2741): computing signature [GET] [/] [] []
2023-01-24T19:13:55.390Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2023-01-24T19:13:55.396Z [CURL DBG] *   Trying 52.216.141.6:443...
2023-01-24T19:13:55.397Z [CURL DBG] * Connected to s3.amazonaws.com (52.216.141.6) port 443 (#4)
2023-01-24T19:13:55.397Z [CURL DBG] * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
2023-01-24T19:13:55.405Z [CURL DBG] * successfully set certificate verify locations:
2023-01-24T19:13:55.405Z [CURL DBG] *  CAfile: /etc/pki/tls/certs/ca-bundle.crt
2023-01-24T19:13:55.405Z [CURL DBG] *  CApath: none
2023-01-24T19:13:55.405Z [CURL DBG] * TLSv1.2 (OUT), TLS header, Certificate Status (22):
2023-01-24T19:13:55.405Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client hello (1):
2023-01-24T19:13:55.407Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server hello (2):
2023-01-24T19:13:55.407Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11):
2023-01-24T19:13:55.409Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
2023-01-24T19:13:55.409Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14):
2023-01-24T19:13:55.410Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
2023-01-24T19:13:55.410Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
2023-01-24T19:13:55.410Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20):
2023-01-24T19:13:55.413Z [CURL DBG] * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
2023-01-24T19:13:55.413Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20):
2023-01-24T19:13:55.413Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
2023-01-24T19:13:55.413Z [CURL DBG] * Server certificate:
2023-01-24T19:13:55.413Z [CURL DBG] *  subject: CN=s3.amazonaws.com
2023-01-24T19:13:55.413Z [CURL DBG] *  start date: Dec  6 00:00:00 2022 GMT
2023-01-24T19:13:55.413Z [CURL DBG] *  expire date: Dec  5 23:59:59 2023 GMT
2023-01-24T19:13:55.413Z [CURL DBG] *  subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com"
2023-01-24T19:13:55.413Z [CURL DBG] *  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01
2023-01-24T19:13:55.413Z [CURL DBG] *  SSL certificate verify ok.
2023-01-24T19:13:55.413Z [CURL DBG] > GET /<bucket_name>/ HTTP/1.1
2023-01-24T19:13:55.413Z [CURL DBG] > Host: s3.amazonaws.com
2023-01-24T19:13:55.413Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL)
2023-01-24T19:13:55.413Z [CURL DBG] > Accept: */*
2023-01-24T19:13:55.413Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20230124/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED
2023-01-24T19:13:55.413Z [CURL DBG] > x-amz-content-sha256: REDACTED
2023-01-24T19:13:55.413Z [CURL DBG] > x-amz-date: 20230124T191355Z
2023-01-24T19:13:55.413Z [CURL DBG] > x-amz-security-token: REDACTED
2023-01-24T19:13:55.413Z [CURL DBG] >
2023-01-24T19:13:55.515Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-01-24T19:13:55.515Z [CURL DBG] < HTTP/1.1 200 OK
2023-01-24T19:13:55.515Z [CURL DBG] < x-amz-id-2: REDACTED
2023-01-24T19:13:55.515Z [CURL DBG] < x-amz-request-id: REDACTED
2023-01-24T19:13:55.515Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:56 GMT
2023-01-24T19:13:55.515Z [CURL DBG] < x-amz-bucket-region: us-east-1
2023-01-24T19:13:55.515Z [CURL DBG] < Content-Type: application/xml
2023-01-24T19:13:55.515Z [CURL DBG] < Transfer-Encoding: chunked
2023-01-24T19:13:55.515Z [CURL DBG] < Server: AmazonS3
2023-01-24T19:13:55.515Z [CURL DBG] <
2023-01-24T19:13:55.518Z [CURL DBG] * Connection #4 to host s3.amazonaws.com left intact
2023-01-24T19:13:55.518Z [INF]       curl.cpp:RequestPerform(2369): HTTP response code 200
2023-01-24T19:13:55.518Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool
2023-01-24T19:13:55.524Z [INF] s3fs.cpp:s3fs_destroy(3958): destroy
2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0)
2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0)
2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0)
2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0)
2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0)

# Another 200, before s3fs_destroy, which I assume is being called because my CTRL+Cs are being detected.

Details about issue

I can't seem to get the PUT /latest/api/token requests in s3fs to return without using (CTRL+C). I think this has something to do with the requests being "chunked", given that pressing CTRL+C yields this message:

^C^C2023-01-24T19:13:44.819Z [CURL DBG] * Signaling end of chunked upload via terminating chunk.

I've tried recreating the curl request that seems to be hanging. In the first snippet, I've kept the Transfer-Encoding: chunked header, and I've removed it in the second snippet:

$ time curl -vvv -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -H "Transfer-Encoding: chunked" -X PUT http://169.254.169.254/latest/api/token
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.69.1
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
> Transfer-Encoding: chunked
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 25 Jan 2023 15:11:46 GMT
< Server: EC2ws
< Connection: close
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Content-Type: text/plain
< Content-Length: 56
<
* Closing connection 0
REDACTED
real    0m30.012s
user    0m0.001s
sys     0m0.007s

$ time curl -vvv -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -X PUT http://169.254.169.254/latest/api/token
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.69.1
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 25 Jan 2023 15:17:47 GMT
< Server: EC2ws
< Connection: close
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Content-Type: text/plain
< Content-Length: 56
<
* Closing connection 0
REDACTED
real    0m0.007s
user    0m0.003s
sys     0m0.003s

Yes, both requests succeeded. However, the chunked request took 30 seconds! Does that sound like a timeout threshold to anyone else?

I suspect this is a duplicate of https://github.com/s3fs-fuse/s3fs-fuse/issues/2029, as adding the options -o imdsv1only -o compat_dir seemed to allow s3fs to mount the bucket and actually see the objects inside of it (-o imdsv1only would mount the bucket, but the directory was empty and the logs showed ENOENT/404 errors until I added -o compat_dir). If I could stop s3fs from sending the chunked encoding header, perhaps -o imdsv1only would not be necessary. Thoughts?

Originally created by @daverysyn on GitHub (Jan 25, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2098 <!-- -------------------------------------------------------------------------- The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD. --------------------------------------------------------------------------- --> ### Additional Information #### Version of s3fs being used (`s3fs --version`) I built s3fs from source but the behavior in the yum 1.91-1 package was identical. ``` $ s3fs --version Amazon Simple Storage Service File System V1.91 (commit:16bc449) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) ``` $ rpm -qi fuse Name : fuse Version : 2.9.2 Release : 11.amzn2 Architecture: x86_64 Install Date: Wed Oct 5 08:55:26 2022 Group : System Environment/Base Size : 222809 License : GPL+ Signature : RSA/SHA256, Thu Dec 6 19:31:53 2018, Key ID 11cf1f95c87f5b1a Source RPM : fuse-2.9.2-11.amzn2.src.rpm Build Date : Fri Nov 16 20:35:39 2018 Build Host : build.amazon.com Relocations : (not relocatable) Packager : Amazon Linux Vendor : Amazon Linux URL : https://github.com/libfuse/libfuse Summary : File System in Userspace (FUSE) utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE userspace tools to mount a FUSE filesystem. ``` #### Kernel information (`uname -r`) ``` $ uname -r 5.10.102-99.473.amzn2.x86_64 ``` #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) ``` $ cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" ``` #### Curl info ``` $ yum list curl libcurl Installed Packages curl.x86_64 7.79.1-7.amzn2.0.1 @amzn2-core libcurl.x86_64 7.79.1-7.amzn2.0.1 @amzn2-core ``` #### How to run s3fs, if applicable ``` $ s3fs <bucket_name> <mount> -o use_path_request_style -o dbglevel=debug -o curldbg -f -s -o iam_role=auto -o ro ``` #### log messages (with my comments) <!-- if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages. --> ``` $ s3fs <bucket_name> <mount> -o use_path_request_style -o dbglevel=debug -o curldbg -f -s -o iam_role=auto -o ro 2023-01-24T19:13:38.073Z [INF] s3fs version 1.91(16bc449) : s3fs -o use_path_request_style -o dbglevel=debug -o curldbg -f -s -o iam_role=auto -o ro <bucket_name> <mount> 2023-01-24T19:13:38.075Z [CRT] s3fs_logger.cpp:LowSetLogLevel(239): change debug level from [CRT] to [DBG] 2023-01-24T19:13:38.075Z [INF] s3fs.cpp:set_mountpoint_attribute(4192): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775) 2023-01-24T19:13:38.076Z [DBG] curl.cpp:InitMimeType(404): Try to load mime types from /etc/mime.types file. 2023-01-24T19:13:38.076Z [DBG] curl.cpp:InitMimeType(408): The old mime types are cleared to load new mime types. 2023-01-24T19:13:38.077Z [INF] curl.cpp:InitMimeType(428): Loaded mime information from /etc/mime.types 2023-01-24T19:13:38.077Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(78): The path to cache top dir is empty, thus not need to check permission. 2023-01-24T19:13:38.077Z [INF] threadpoolman.cpp:StopThreads(195): Any threads are running now, then nothing to do. 2023-01-24T19:13:38.078Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-24T19:13:38.078Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-24T19:13:38.078Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-24T19:13:38.078Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-24T19:13:38.078Z [INF] threadpoolman.cpp:Worker(76): Start worker thread in ThreadPoolMan. 2023-01-24T19:13:38.081Z [CRT] s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function. 2023-01-24T19:13:38.081Z [INF] s3fs.cpp:s3fs_init(3914): init v1.91(commit:16bc449) with OpenSSL, credential-library(built-in) 2023-01-24T19:13:38.081Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-01-24T19:13:38.081Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/api/token 2023-01-24T19:13:38.081Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-24T19:13:38.082Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2023-01-24T19:13:38.082Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2023-01-24T19:13:38.082Z [CURL DBG] > Host: 169.254.169.254 2023-01-24T19:13:38.082Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-24T19:13:38.082Z [CURL DBG] > Accept: */* 2023-01-24T19:13:38.082Z [CURL DBG] > Transfer-Encoding: chunked 2023-01-24T19:13:38.082Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2023-01-24T19:13:38.082Z [CURL DBG] > # Output hangs here and doesn't proceed. # Add visual spacing by pressing enter a few times (doesn't alter behavior, as far as I can tell) # CTRL+C twice seems to move things along (Pressing CTRL+C once only works if I don't use Enter to create visual spacing) # It looks to me like this got back a 200, so why did the request hang? ^C^C2023-01-24T19:13:44.819Z [CURL DBG] * Signaling end of chunked upload via terminating chunk. 2023-01-24T19:13:44.819Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-24T19:13:44.823Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-24T19:13:44.823Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:40 GMT 2023-01-24T19:13:44.823Z [CURL DBG] < Server: EC2ws 2023-01-24T19:13:44.823Z [CURL DBG] < Connection: close 2023-01-24T19:13:44.823Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-01-24T19:13:44.823Z [CURL DBG] < Content-Type: text/plain 2023-01-24T19:13:44.823Z [CURL DBG] < Content-Length: 56 2023-01-24T19:13:44.823Z [CURL DBG] < 2023-01-24T19:13:44.823Z [CURL DBG] * Closing connection 0 2023-01-24T19:13:44.823Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-24T19:13:44.823Z [INF] s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to REDACTED 2023-01-24T19:13:44.823Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-01-24T19:13:44.823Z [INF] curl.cpp:GetIAMRoleFromMetaData(3006): Get IAM Role name 2023-01-24T19:13:44.823Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-01-24T19:13:44.823Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2023-01-24T19:13:44.823Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache 2023-01-24T19:13:44.823Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-24T19:13:44.823Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1) 2023-01-24T19:13:44.823Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1 2023-01-24T19:13:44.823Z [CURL DBG] > Host: 169.254.169.254 2023-01-24T19:13:44.823Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-24T19:13:44.823Z [CURL DBG] > Accept: */* 2023-01-24T19:13:44.823Z [CURL DBG] > X-aws-ec2-metadata-token: REDACTED 2023-01-24T19:13:44.823Z [CURL DBG] > 2023-01-24T19:13:44.825Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-24T19:13:44.825Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-24T19:13:44.825Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:44 GMT 2023-01-24T19:13:44.825Z [CURL DBG] < Accept-Ranges: none 2023-01-24T19:13:44.825Z [CURL DBG] < Server: EC2ws 2023-01-24T19:13:44.825Z [CURL DBG] < Connection: close 2023-01-24T19:13:44.825Z [CURL DBG] < Last-Modified: Tue, 24 Jan 2023 18:45:22 GMT 2023-01-24T19:13:44.825Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21596 2023-01-24T19:13:44.825Z [CURL DBG] < Content-Type: text/plain 2023-01-24T19:13:44.825Z [CURL DBG] < Content-Length: 35 2023-01-24T19:13:44.825Z [CURL DBG] < 2023-01-24T19:13:44.825Z [CURL DBG] * Closing connection 1 2023-01-24T19:13:44.825Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-24T19:13:44.825Z [INF] s3fs_cred.cpp:SetIAMRoleFromMetaData(582): IAM role name response = "<role>" 2023-01-24T19:13:44.825Z [INF] s3fs_cred.cpp:LoadIAMRoleFromMetaData(543): loaded IAM role name = <role> 2023-01-24T19:13:44.825Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-01-24T19:13:44.825Z [INF] s3fs.cpp:s3fs_check_service(4058): check services. 2023-01-24T19:13:44.825Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1152): IAM Access Token refreshing... 2023-01-24T19:13:44.825Z [INF] s3fs_cred.cpp:GetIAMCredentialsURL(374): [IAM role=<role>] 2023-01-24T19:13:44.825Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-01-24T19:13:44.825Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/api/token 2023-01-24T19:13:44.825Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache 2023-01-24T19:13:44.825Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-24T19:13:44.825Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#2) 2023-01-24T19:13:44.825Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2023-01-24T19:13:44.825Z [CURL DBG] > Host: 169.254.169.254 2023-01-24T19:13:44.825Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-24T19:13:44.825Z [CURL DBG] > Accept: */* 2023-01-24T19:13:44.825Z [CURL DBG] > Transfer-Encoding: chunked 2023-01-24T19:13:44.825Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2023-01-24T19:13:44.825Z [CURL DBG] > # This request also hangs # Press enter a few times to create space # CTRL+C twice to get things moving again. Again, we see that a 200 came back on this request, so why did it hang up? ^C^C2023-01-24T19:13:55.386Z [CURL DBG] * Signaling end of chunked upload via terminating chunk. 2023-01-24T19:13:55.386Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-24T19:13:55.387Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-24T19:13:55.387Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:54 GMT 2023-01-24T19:13:55.387Z [CURL DBG] < Server: EC2ws 2023-01-24T19:13:55.387Z [CURL DBG] < Connection: close 2023-01-24T19:13:55.387Z [CURL DBG] < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 2023-01-24T19:13:55.387Z [CURL DBG] < Content-Type: text/plain 2023-01-24T19:13:55.387Z [CURL DBG] < Content-Length: 56 2023-01-24T19:13:55.387Z [CURL DBG] < 2023-01-24T19:13:55.387Z [CURL DBG] * Closing connection 2 2023-01-24T19:13:55.387Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-24T19:13:55.387Z [INF] s3fs_cred.cpp:SetIAMv2APIToken(447): Setting AWS IMDSv2 API token to REDACTED 2023-01-24T19:13:55.387Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-01-24T19:13:55.387Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-01-24T19:13:55.387Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/<role> 2023-01-24T19:13:55.387Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache 2023-01-24T19:13:55.387Z [CURL DBG] * Trying 169.254.169.254:80... 2023-01-24T19:13:55.388Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#3) 2023-01-24T19:13:55.388Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/<role> HTTP/1.1 2023-01-24T19:13:55.388Z [CURL DBG] > Host: 169.254.169.254 2023-01-24T19:13:55.388Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-24T19:13:55.388Z [CURL DBG] > Accept: */* 2023-01-24T19:13:55.388Z [CURL DBG] > X-aws-ec2-metadata-token: REDACTED 2023-01-24T19:13:55.388Z [CURL DBG] > 2023-01-24T19:13:55.390Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-24T19:13:55.390Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-24T19:13:55.390Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:55 GMT 2023-01-24T19:13:55.390Z [CURL DBG] < Content-Type: text/plain 2023-01-24T19:13:55.390Z [CURL DBG] < Content-Length: 958 2023-01-24T19:13:55.390Z [CURL DBG] < Server: Jetty(9.4.z-SNAPSHOT) 2023-01-24T19:13:55.390Z [CURL DBG] < 2023-01-24T19:13:55.390Z [CURL DBG] * Connection #3 to host 169.254.169.254 left intact 2023-01-24T19:13:55.390Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-24T19:13:55.390Z [INF] s3fs_cred.cpp:SetIAMCredentials(550): IAM credential response = "{"AccessKeyId":"REDACTED","SecretAccessKey":"REDACTED","Token":"REDACTED","Expiration":"2023-01-24T20:13:30Z","Code":"Success","Message":null}" 2023-01-24T19:13:55.390Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-01-24T19:13:55.390Z [INF] s3fs_cred.cpp:CheckIAMCredentialUpdate(1166): IAM Access Token refreshed 2023-01-24T19:13:55.390Z [INF] curl.cpp:CheckBucket(3519): check a bucket. 2023-01-24T19:13:55.390Z [DBG] curl_handlerpool.cpp:GetHandler(79): Get handler from pool: rest = 31 2023-01-24T19:13:55.390Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/<bucket_name>/ 2023-01-24T19:13:55.390Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://s3.amazonaws.com/<bucket_name>/ 2023-01-24T19:13:55.390Z [DBG] curl.cpp:RequestPerform(2334): connecting to URL https://s3.amazonaws.com/<bucket_name>/ 2023-01-24T19:13:55.390Z [INF] curl.cpp:insertV4Headers(2741): computing signature [GET] [/] [] [] 2023-01-24T19:13:55.390Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com 2023-01-24T19:13:55.396Z [CURL DBG] * Trying 52.216.141.6:443... 2023-01-24T19:13:55.397Z [CURL DBG] * Connected to s3.amazonaws.com (52.216.141.6) port 443 (#4) 2023-01-24T19:13:55.397Z [CURL DBG] * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH 2023-01-24T19:13:55.405Z [CURL DBG] * successfully set certificate verify locations: 2023-01-24T19:13:55.405Z [CURL DBG] * CAfile: /etc/pki/tls/certs/ca-bundle.crt 2023-01-24T19:13:55.405Z [CURL DBG] * CApath: none 2023-01-24T19:13:55.405Z [CURL DBG] * TLSv1.2 (OUT), TLS header, Certificate Status (22): 2023-01-24T19:13:55.405Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client hello (1): 2023-01-24T19:13:55.407Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server hello (2): 2023-01-24T19:13:55.407Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Certificate (11): 2023-01-24T19:13:55.409Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server key exchange (12): 2023-01-24T19:13:55.409Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Server finished (14): 2023-01-24T19:13:55.410Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): 2023-01-24T19:13:55.410Z [CURL DBG] * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): 2023-01-24T19:13:55.410Z [CURL DBG] * TLSv1.2 (OUT), TLS handshake, Finished (20): 2023-01-24T19:13:55.413Z [CURL DBG] * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): 2023-01-24T19:13:55.413Z [CURL DBG] * TLSv1.2 (IN), TLS handshake, Finished (20): 2023-01-24T19:13:55.413Z [CURL DBG] * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 2023-01-24T19:13:55.413Z [CURL DBG] * Server certificate: 2023-01-24T19:13:55.413Z [CURL DBG] * subject: CN=s3.amazonaws.com 2023-01-24T19:13:55.413Z [CURL DBG] * start date: Dec 6 00:00:00 2022 GMT 2023-01-24T19:13:55.413Z [CURL DBG] * expire date: Dec 5 23:59:59 2023 GMT 2023-01-24T19:13:55.413Z [CURL DBG] * subjectAltName: host "s3.amazonaws.com" matched cert's "s3.amazonaws.com" 2023-01-24T19:13:55.413Z [CURL DBG] * issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01 2023-01-24T19:13:55.413Z [CURL DBG] * SSL certificate verify ok. 2023-01-24T19:13:55.413Z [CURL DBG] > GET /<bucket_name>/ HTTP/1.1 2023-01-24T19:13:55.413Z [CURL DBG] > Host: s3.amazonaws.com 2023-01-24T19:13:55.413Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash 16bc449; OpenSSL) 2023-01-24T19:13:55.413Z [CURL DBG] > Accept: */* 2023-01-24T19:13:55.413Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20230124/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=REDACTED 2023-01-24T19:13:55.413Z [CURL DBG] > x-amz-content-sha256: REDACTED 2023-01-24T19:13:55.413Z [CURL DBG] > x-amz-date: 20230124T191355Z 2023-01-24T19:13:55.413Z [CURL DBG] > x-amz-security-token: REDACTED 2023-01-24T19:13:55.413Z [CURL DBG] > 2023-01-24T19:13:55.515Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-01-24T19:13:55.515Z [CURL DBG] < HTTP/1.1 200 OK 2023-01-24T19:13:55.515Z [CURL DBG] < x-amz-id-2: REDACTED 2023-01-24T19:13:55.515Z [CURL DBG] < x-amz-request-id: REDACTED 2023-01-24T19:13:55.515Z [CURL DBG] < Date: Tue, 24 Jan 2023 19:13:56 GMT 2023-01-24T19:13:55.515Z [CURL DBG] < x-amz-bucket-region: us-east-1 2023-01-24T19:13:55.515Z [CURL DBG] < Content-Type: application/xml 2023-01-24T19:13:55.515Z [CURL DBG] < Transfer-Encoding: chunked 2023-01-24T19:13:55.515Z [CURL DBG] < Server: AmazonS3 2023-01-24T19:13:55.515Z [CURL DBG] < 2023-01-24T19:13:55.518Z [CURL DBG] * Connection #4 to host s3.amazonaws.com left intact 2023-01-24T19:13:55.518Z [INF] curl.cpp:RequestPerform(2369): HTTP response code 200 2023-01-24T19:13:55.518Z [DBG] curl_handlerpool.cpp:ReturnHandler(101): Return handler to pool 2023-01-24T19:13:55.524Z [INF] s3fs.cpp:s3fs_destroy(3958): destroy 2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0) 2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0) 2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0) 2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0) 2023-01-24T19:13:55.524Z [DBG] threadpoolman.cpp:StopThreads(212): succeed pthread_join - return code(0) # Another 200, before s3fs_destroy, which I assume is being called because my CTRL+Cs are being detected. ``` ### Details about issue <!-- Please describe the content of the issue in detail. --> I can't seem to get the `PUT /latest/api/token` requests in s3fs to return without using (CTRL+C). I think this has something to do with the requests being "chunked", given that pressing CTRL+C yields this message: ``` ^C^C2023-01-24T19:13:44.819Z [CURL DBG] * Signaling end of chunked upload via terminating chunk. ``` I've tried recreating the curl request that seems to be hanging. In the first snippet, I've kept the `Transfer-Encoding: chunked` header, and I've removed it in the second snippet: ``` $ time curl -vvv -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -H "Transfer-Encoding: chunked" -X PUT http://169.254.169.254/latest/api/token * Trying 169.254.169.254:80... * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) > PUT /latest/api/token HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.69.1 > Accept: */* > X-aws-ec2-metadata-token-ttl-seconds: 21600 > Transfer-Encoding: chunked > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Wed, 25 Jan 2023 15:11:46 GMT < Server: EC2ws < Connection: close < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 < Content-Type: text/plain < Content-Length: 56 < * Closing connection 0 REDACTED real 0m30.012s user 0m0.001s sys 0m0.007s ``` ``` $ time curl -vvv -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" -X PUT http://169.254.169.254/latest/api/token * Trying 169.254.169.254:80... * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) > PUT /latest/api/token HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.69.1 > Accept: */* > X-aws-ec2-metadata-token-ttl-seconds: 21600 > * Mark bundle as not supporting multiuse < HTTP/1.1 200 OK < Date: Wed, 25 Jan 2023 15:17:47 GMT < Server: EC2ws < Connection: close < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 < Content-Type: text/plain < Content-Length: 56 < * Closing connection 0 REDACTED real 0m0.007s user 0m0.003s sys 0m0.003s ``` Yes, both requests succeeded. However, the chunked request took 30 seconds! Does that sound like a timeout threshold to anyone else? I suspect this is a duplicate of https://github.com/s3fs-fuse/s3fs-fuse/issues/2029, as adding the options `-o imdsv1only -o compat_dir` seemed to allow s3fs to mount the bucket and actually see the objects inside of it (`-o imdsv1only` would mount the bucket, but the directory was empty and the logs showed ENOENT/404 errors until I added `-o compat_dir`). If I could stop s3fs from sending the chunked encoding header, perhaps `-o imdsv1only` would not be necessary. Thoughts?
kerem closed this issue 2026-03-04 01:51:06 +03:00
kerem commented 2026-03-04 01:51:07 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 29, 2023):

@daverysyn Thanks for the details of the bug about deadlock.
I found the cause of this and fixed it.
I'm posting a PR #2103 for the fix, pelase wait until it is merged.

<!-- gh-comment-id:1407612830 --> @ggtakec commented on GitHub (Jan 29, 2023): @daverysyn Thanks for the details of the bug about deadlock. I found the cause of this and fixed it. I'm posting a PR #2103 for the fix, pelase wait until it is merged.
kerem commented 2026-03-04 01:51:07 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 1, 2023):

#2103 has been merged.
I believe this bug has been fixed.
I will close this issue, but if you still have problems please reopen or post a new issue.

<!-- gh-comment-id:1412183152 --> @ggtakec commented on GitHub (Feb 1, 2023): #2103 has been merged. I believe this bug has been fixed. I will close this issue, but if you still have problems please reopen or post a new issue.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1068
No description provided.