[GH-ISSUE #1765] Receiving 417 Expectation Failed when querying for IMDSv2 token #909

New issue

Closed

opened 2026-03-04 01:49:49 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:49:49 +03:00

Owner

Originally created by @kamilJ96 on GitHub (Sep 8, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1765

Additional Information

When using IMDSv2, the first request that occurs is a PUT request to /latest/api/token in order to retrieve a valid access token. By default, curl adds an Expect: 100-continue header in the request, and AWS responds with a 417 Expectation Failed.

Now, curl will automatically retry this request without the Expect header, however it only does this for requests that have a body, which this request does not (see HERE), thus this request fails and s3fs defaults to using IMDSv1. If IMDSv1 is disabled on the instance, then the whole thing will fail.

Version of s3fs being used (s3fs --version)

V1.90 (commit:9d1552a)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2
(curl: 7.76.1)

Kernel information (uname -r)

Amazon Linux 2

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Sep  8 09:56:30 :      Get IAM Role name
Sep  8 09:56:30 : *   Trying 169.254.169.254:80...
Sep  8 09:56:30 : * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
Sep  8 09:56:30 : > PUT /latest/api/token HTTP/1.1
Sep  8 09:56:30 : > Host: 169.254.169.254
Sep  8 09:56:30 : > User-Agent: s3fs/1.90 (commit hash 9d1552a; OpenSSL)
Sep  8 09:56:30 : > Accept: */*
Sep  8 09:56:30 : > Transfer-Encoding: chunked
Sep  8 09:56:30 : > X-aws-ec2-metadata-token-ttl-seconds: 21600
Sep  8 09:56:30 : > Expect: 100-continue <---------------  ❌  BAD HEADER ❌
Sep  8 09:56:30 : >
Sep  8 09:56:30 : * Mark bundle as not supporting multiuse
Sep  8 09:56:30 : * HTTP 1.0, assume close after body
Sep  8 09:56:30 : < HTTP/1.0 417 Expectation Failed
Sep  8 09:56:30 : < Content-Type: text/html
Sep  8 09:56:30 : < Content-Length: 363
Sep  8 09:56:30 : < Connection: close
Sep  8 09:56:30 : < Date: Wed, 08 Sep 2021 09:56:30 GMT
Sep  8 09:56:30 : < Server: EC2ws
Sep  8 09:56:30 : <
Sep  8 09:56:30 : * Closing connection 0
Sep  8 09:56:30 : curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>#012<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"#012         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">#012<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">#012 <head>#012  <title>417 - Expectation Failed</title>#012 </head>#012 <body>#012  <h1>417 - Expectation Failed</h1>#012 </body>#012</html>
Sep  8 09:56:30 : curl.cpp:LoadIAMRoleFromMetaData(3212): AWS IMDSv2 token retrieval failed: -5

Originally created by @kamilJ96 on GitHub (Sep 8, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1765 ### Additional Information When using IMDSv2, the first request that occurs is a PUT request to `/latest/api/token` in order to retrieve a valid access token. By default, curl adds an `Expect: 100-continue` header in the request, and AWS responds with a `417 Expectation Failed`. Now, curl will automatically retry this request without the `Expect` header, however it only does this for requests that have a body, which this request does not (see [HERE](https://github.com/curl/curl/pull/4964)), thus this request fails and s3fs defaults to using IMDSv1. If IMDSv1 is disabled on the instance, then the whole thing will fail. #### Version of s3fs being used (s3fs --version) V1.90 (commit:9d1552a) #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.2 (curl: 7.76.1) #### Kernel information (uname -r) Amazon Linux 2 #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` Sep 8 09:56:30 : Get IAM Role name Sep 8 09:56:30 : * Trying 169.254.169.254:80... Sep 8 09:56:30 : * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) Sep 8 09:56:30 : > PUT /latest/api/token HTTP/1.1 Sep 8 09:56:30 : > Host: 169.254.169.254 Sep 8 09:56:30 : > User-Agent: s3fs/1.90 (commit hash 9d1552a; OpenSSL) Sep 8 09:56:30 : > Accept: */* Sep 8 09:56:30 : > Transfer-Encoding: chunked Sep 8 09:56:30 : > X-aws-ec2-metadata-token-ttl-seconds: 21600 Sep 8 09:56:30 : > Expect: 100-continue <--------------- ❌ BAD HEADER ❌ Sep 8 09:56:30 : > Sep 8 09:56:30 : * Mark bundle as not supporting multiuse Sep 8 09:56:30 : * HTTP 1.0, assume close after body Sep 8 09:56:30 : < HTTP/1.0 417 Expectation Failed Sep 8 09:56:30 : < Content-Type: text/html Sep 8 09:56:30 : < Content-Length: 363 Sep 8 09:56:30 : < Connection: close Sep 8 09:56:30 : < Date: Wed, 08 Sep 2021 09:56:30 GMT Sep 8 09:56:30 : < Server: EC2ws Sep 8 09:56:30 : < Sep 8 09:56:30 : * Closing connection 0 Sep 8 09:56:30 : curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>#012<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"#012 "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">#012<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">#012 <head>#012 <title>417 - Expectation Failed</title>#012 </head>#012 <body>#012 <h1>417 - Expectation Failed</h1>#012 </body>#012</html> Sep 8 09:56:30 : curl.cpp:LoadIAMRoleFromMetaData(3212): AWS IMDSv2 token retrieval failed: -5 ```

kerem closed this issue

2026-03-04 01:49:49 +03:00

kerem commented

2026-03-04 01:49:50 +03:00

Author

Owner

@nmeyerhans commented on GitHub (Sep 8, 2021):

For the record, this seems to only impact Xen-based EC2 instances; I have tested on t4g instances (Nitro-based arm64 instances) and the IMDS web server does respond with a token as expected, and does not return a 417 error. However, testing on t2 instances (Xen-based), does reproduce the problem as reported.

@nmeyerhans commented on GitHub (Sep 8, 2021): For the record, this seems to only impact Xen-based EC2 instances; I have tested on t4g instances (Nitro-based arm64 instances) and the IMDS web server does respond with a token as expected, and does not return a 417 error. However, testing on t2 instances (Xen-based), does reproduce the problem as reported.

kerem commented

2026-03-04 01:49:50 +03:00

Author

Owner

@kamilJ96 commented on GitHub (Sep 8, 2021):

@nmeyerhans Thanks AWS - I was wondering why this problem wasn't more prevalent but I guess it just needed a perfect storm of conditions before coming up 😄

@gaul do you have an estimate for the next release?

Thanks to both of you as well for help/approval! The s3fs code is really beautiful

@kamilJ96 commented on GitHub (Sep 8, 2021): @nmeyerhans Thanks AWS - I was wondering why this problem wasn't more prevalent but I guess it just needed a perfect storm of conditions before coming up 😄 @gaul do you have an estimate for the next release? Thanks to both of you as well for help/approval! The s3fs code is really beautiful

kerem commented

2026-03-04 01:49:50 +03:00

Author

Owner

@gaul commented on GitHub (Sep 8, 2021):

Probably February since s3fs aims for a release every six months and we just released 1.90.

@gaul commented on GitHub (Sep 8, 2021): Probably February since s3fs aims for a release every six months and we just released 1.90.

kerem referenced this issue

2026-03-04 02:01:42 +03:00

[PR #909] [MERGED] Ignore after period character of floating point in x-amz-meta-mtime #1693