[GH-ISSUE #1581] Mount hangs after initial request receives HTTP 100 Continue #829

Closed
opened 2026-03-04 01:49:09 +03:00 by kerem · 6 comments
Owner

Originally created by @mun1r0b0t on GitHub (Feb 19, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1581

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

$ s3fs --version
Amazon Simple Storage Service File System V1.88 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

$ dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 140
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: amd64
Version: 2.9.9-1+deb10u1
Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-1+deb10u1), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
 /etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
 Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
 export a virtual filesystem to the Linux kernel. It also aims to provide a
 secure method for non privileged users to create and mount their own filesystem
 implementations.
Homepage: https://github.com/libfuse/libfuse/wiki

Kernel information (uname -r)

$ uname -r
5.9.0-1.el7.elrepo.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

$ uname -r
5.9.0-1.el7.elrepo.x86_64
bon@db4400b7cfc9:/ninja$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

s3fs command line used, if applicable

s3fs -f -d ${BUCKET} ${MOUNT_POINT} -o iam_role=$IAM_ROLE -o nonempty -o use_cache=.s3fs-cache -o del_cache -o stat_cache_expire=900 -o enable_noobj_cache -o umask=022 -o allow_other -o curldbg -o dbglevel=info

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

[CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from [CRT] to [INF] 
[INF]     s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=1001, gid=1001) - MountPoint(uid=1001, gid=1001, mode=40755)
[INF] curl.cpp:InitMimeType(438): Loaded mime information from /etc/mime.types
[INF] s3fs.cpp:s3fs_init(3300): init v1.88(commit:unknown) with OpenSSL
[ERR] s3fs_util.cpp:delete_files_in_dir(300): could not open dir(.s3fs-cache/.<bucket-name-redacted>.stat) - errno(2)
[INF] s3fs.cpp:s3fs_check_service(3416): check services.
[INF] curl.cpp:CheckIAMCredentialUpdate(1721): IAM Access Token refreshing...
[INF]       curl.cpp:GetIAMCredentials(2742): [IAM role=<role-name-redacted>]
[CURL DBG] * Expire in 0 ms for 6 (transfer 0x562eba7cb200)
[CURL DBG] * Expire in 300000 ms for 2 (transfer 0x562eba7cb200)
[CURL DBG] *   Trying 169.254.169.254...
[CURL DBG] * TCP_NODELAY set
[CURL DBG] * Expire in 200 ms for 4 (transfer 0x562eba7cb200)
[CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
[CURL DBG] > PUT /latest/api/token HTTP/1.1
[CURL DBG] > Host: 169.254.169.254
[CURL DBG] > User-Agent: s3fs/1.88 (commit hash unknown; OpenSSL)
[CURL DBG] > Accept: */*
[CURL DBG] > Transfer-Encoding: chunked
[CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
[CURL DBG] > Expect: 100-continue
[CURL DBG] > 
[CURL DBG] * Expire in 1000 ms for 0 (transfer 0x562eba7cb200)
[CURL DBG] < HTTP/1.1 100 Continue

Details about issue

S3 bucket is mounted to a mount point inside a docker image. The output of the commands above is from within the image. s3fs is compiled from source as part of the docker build after checking out v1.88 tag from the git repo.

Basically, the mount hangs after the output listed above. In this state, I can see the mount listed in the output of mount. But if I try to run ls or any other command on the mount point it also hangs. The only way to stop this is with kill -9. Even hitting Ctrl+C doesn't terminate the process.

I noticed when leaving the container up overnight, that the command eventually moves forward, but the same issue then occurs when trying to read from the files.

Please let me know if you need more info or any suggestions on what's wrong.

Originally created by @mun1r0b0t on GitHub (Feb 19, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1581 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) ``` $ s3fs --version Amazon Simple Storage Service File System V1.88 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` $ dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 140 Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Architecture: amd64 Version: 2.9.9-1+deb10u1 Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-1+deb10u1), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Homepage: https://github.com/libfuse/libfuse/wiki ``` #### Kernel information (uname -r) ``` $ uname -r 5.9.0-1.el7.elrepo.x86_64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` $ uname -r 5.9.0-1.el7.elrepo.x86_64 bon@db4400b7cfc9:/ninja$ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` #### s3fs command line used, if applicable ``` s3fs -f -d ${BUCKET} ${MOUNT_POINT} -o iam_role=$IAM_ROLE -o nonempty -o use_cache=.s3fs-cache -o del_cache -o stat_cache_expire=900 -o enable_noobj_cache -o umask=022 -o allow_other -o curldbg -o dbglevel=info ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` [CRT] s3fs_logger.cpp:LowSetLogLevel(201): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(3989): PROC(uid=1001, gid=1001) - MountPoint(uid=1001, gid=1001, mode=40755) [INF] curl.cpp:InitMimeType(438): Loaded mime information from /etc/mime.types [INF] s3fs.cpp:s3fs_init(3300): init v1.88(commit:unknown) with OpenSSL [ERR] s3fs_util.cpp:delete_files_in_dir(300): could not open dir(.s3fs-cache/.<bucket-name-redacted>.stat) - errno(2) [INF] s3fs.cpp:s3fs_check_service(3416): check services. [INF] curl.cpp:CheckIAMCredentialUpdate(1721): IAM Access Token refreshing... [INF] curl.cpp:GetIAMCredentials(2742): [IAM role=<role-name-redacted>] [CURL DBG] * Expire in 0 ms for 6 (transfer 0x562eba7cb200) [CURL DBG] * Expire in 300000 ms for 2 (transfer 0x562eba7cb200) [CURL DBG] * Trying 169.254.169.254... [CURL DBG] * TCP_NODELAY set [CURL DBG] * Expire in 200 ms for 4 (transfer 0x562eba7cb200) [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) [CURL DBG] > PUT /latest/api/token HTTP/1.1 [CURL DBG] > Host: 169.254.169.254 [CURL DBG] > User-Agent: s3fs/1.88 (commit hash unknown; OpenSSL) [CURL DBG] > Accept: */* [CURL DBG] > Transfer-Encoding: chunked [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 [CURL DBG] > Expect: 100-continue [CURL DBG] > [CURL DBG] * Expire in 1000 ms for 0 (transfer 0x562eba7cb200) [CURL DBG] < HTTP/1.1 100 Continue ``` ### Details about issue S3 bucket is mounted to a mount point inside a docker image. The output of the commands above is from within the image. s3fs is compiled from source as part of the docker build after checking out v1.88 tag from the git repo. Basically, the mount hangs after the output listed above. In this state, I can see the mount listed in the output of `mount`. But if I try to run `ls` or any other command on the mount point it also hangs. The only way to stop this is with `kill -9`. Even hitting Ctrl+C doesn't terminate the process. I noticed when leaving the container up overnight, that the command eventually moves forward, but the same issue then occurs when trying to read from the files. Please let me know if you need more info or any suggestions on what's wrong.
kerem closed this issue 2026-03-04 01:49:09 +03:00
Author
Owner

@adamstrawson commented on GitHub (Mar 1, 2021):

We're encountering this same issue too, under the same version.

Only difference here is the OS version, below if it helps with troubleshooting;

uname -r
4.14.209-160.339.amzn2.x86_64
cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.12.4
PRETTY_NAME="Alpine Linux v3.12"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
<!-- gh-comment-id:788226973 --> @adamstrawson commented on GitHub (Mar 1, 2021): We're encountering this same issue too, under the same version. Only difference here is the OS version, below if it helps with troubleshooting; ``` uname -r 4.14.209-160.339.amzn2.x86_64 ``` ``` cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.12.4 PRETTY_NAME="Alpine Linux v3.12" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://bugs.alpinelinux.org/" ```
Author
Owner

@gaul commented on GitHub (Mar 7, 2021):

Which S3 server are you testing with, AWS or another implementation? If the latter I suspect some buggy behavior in the server since AWS works for me. libcurl does not have an obvious way to disable Expected: 100-continue although they discussed disabling it by default a few years ago. Can you try adding:

requestHeaders = curl_slist_sort_insert(requestHeaders, "Expect", "");

to S3fsCurl::PutHeadRequest? Maybe this is something s3fs should do by default or at least add a flag.

<!-- gh-comment-id:792189941 --> @gaul commented on GitHub (Mar 7, 2021): Which S3 server are you testing with, AWS or another implementation? If the latter I suspect some buggy behavior in the server since AWS works for me. libcurl does not have an obvious way to disable `Expected: 100-continue` although they [discussed disabling it by default](https://curl.se/mail/lib-2017-07/0013.html) a few years ago. Can you try adding: ```c requestHeaders = curl_slist_sort_insert(requestHeaders, "Expect", ""); ``` to `S3fsCurl::PutHeadRequest`? Maybe this is something s3fs should do by default or at least add a flag.
Author
Owner

@adamstrawson commented on GitHub (Mar 7, 2021):

@gaul For me, we were using AWS (Region: eu-west-1).

Wasn't doing anything else out the ordinary for a standard setup based on the documentation.

<!-- gh-comment-id:792363745 --> @adamstrawson commented on GitHub (Mar 7, 2021): @gaul For me, we were using AWS (Region: `eu-west-1`). Wasn't doing anything else out the ordinary for a standard setup based on the documentation.
Author
Owner

@giafar commented on GitHub (Jun 15, 2021):

Same behaviour here.

S3FS package is installed via amazon-linux-extras install epel ; yum install s3fs-fuse and the docker image is an amazonlinux:latest.
Any help is appreciated

OS

$ cat /etc/os-release 
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

Kernel version

$ uname -r
4.14.209-160.339.amzn2.x86_64

s3fs version

$ s3fs --version
Amazon Simple Storage Service File System V1.89 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

s3fs mount command

$ s3fs mybucket  -o iam_role=myrole -o url=https://s3-eu-central-1.amazonaws.com \
  -o endpoint=eu-central-1 -o dbglevel=info -o curldbg -o allow_other -o use_cache=/tmp -f /mnt

2021-06-15T18:25:49.743Z [CRT] s3fs_logger.cpp:LowSetLogLevel(219): change debug level from [CRT] to [INF] 
2021-06-15T18:25:49.743Z [INF]     s3fs.cpp:set_mountpoint_attribute(4020): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
2021-06-15T18:25:49.744Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types
2021-06-15T18:25:49.745Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with OpenSSL
2021-06-15T18:25:49.745Z [INF] s3fs.cpp:s3fs_check_service(3447): check services.
2021-06-15T18:25:49.745Z [INF] curl.cpp:CheckIAMCredentialUpdate(1741): IAM Access Token refreshing...
2021-06-15T18:25:49.745Z [INF]       curl.cpp:GetIAMCredentials(2784): [IAM role=****************************]
2021-06-15T18:25:49.745Z [CURL DBG] *   Trying 169.254.169.254...
2021-06-15T18:25:49.745Z [CURL DBG] * TCP_NODELAY set
2021-06-15T18:25:49.746Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2021-06-15T18:25:49.746Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2021-06-15T18:25:49.746Z [CURL DBG] > Host: 169.254.169.254
2021-06-15T18:25:49.746Z [CURL DBG] > User-Agent: s3fs/1.89 (commit hash unknown; OpenSSL)
2021-06-15T18:25:49.746Z [CURL DBG] > Accept: */*
2021-06-15T18:25:49.746Z [CURL DBG] > Transfer-Encoding: chunked
2021-06-15T18:25:49.746Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2021-06-15T18:25:49.746Z [CURL DBG] > Expect: 100-continue
2021-06-15T18:25:49.746Z [CURL DBG] > 
2021-06-15T18:25:49.746Z [CURL DBG] < HTTP/1.1 100 Continue

<!-- gh-comment-id:861736500 --> @giafar commented on GitHub (Jun 15, 2021): Same behaviour here. S3FS package is installed via `amazon-linux-extras install epel ; yum install s3fs-fuse` and the docker image is an `amazonlinux:latest`. Any help is appreciated **OS** ```bash $ cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" ``` **Kernel version** ```bash $ uname -r 4.14.209-160.339.amzn2.x86_64 ``` **s3fs version** ```bash $ s3fs --version Amazon Simple Storage Service File System V1.89 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` **s3fs mount command** ```bash $ s3fs mybucket -o iam_role=myrole -o url=https://s3-eu-central-1.amazonaws.com \ -o endpoint=eu-central-1 -o dbglevel=info -o curldbg -o allow_other -o use_cache=/tmp -f /mnt 2021-06-15T18:25:49.743Z [CRT] s3fs_logger.cpp:LowSetLogLevel(219): change debug level from [CRT] to [INF] 2021-06-15T18:25:49.743Z [INF] s3fs.cpp:set_mountpoint_attribute(4020): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) 2021-06-15T18:25:49.744Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types 2021-06-15T18:25:49.745Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with OpenSSL 2021-06-15T18:25:49.745Z [INF] s3fs.cpp:s3fs_check_service(3447): check services. 2021-06-15T18:25:49.745Z [INF] curl.cpp:CheckIAMCredentialUpdate(1741): IAM Access Token refreshing... 2021-06-15T18:25:49.745Z [INF] curl.cpp:GetIAMCredentials(2784): [IAM role=****************************] 2021-06-15T18:25:49.745Z [CURL DBG] * Trying 169.254.169.254... 2021-06-15T18:25:49.745Z [CURL DBG] * TCP_NODELAY set 2021-06-15T18:25:49.746Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2021-06-15T18:25:49.746Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2021-06-15T18:25:49.746Z [CURL DBG] > Host: 169.254.169.254 2021-06-15T18:25:49.746Z [CURL DBG] > User-Agent: s3fs/1.89 (commit hash unknown; OpenSSL) 2021-06-15T18:25:49.746Z [CURL DBG] > Accept: */* 2021-06-15T18:25:49.746Z [CURL DBG] > Transfer-Encoding: chunked 2021-06-15T18:25:49.746Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2021-06-15T18:25:49.746Z [CURL DBG] > Expect: 100-continue 2021-06-15T18:25:49.746Z [CURL DBG] > 2021-06-15T18:25:49.746Z [CURL DBG] < HTTP/1.1 100 Continue ```
Author
Owner

@jvassev commented on GitHub (Jul 10, 2021):

I can confirm the same with V1.89 on AmazonLinux. I performed the same request with curl to see if it can handle AWS's implementation as doubted in https://github.com/s3fs-fuse/s3fs-fuse/issues/1581#issuecomment-792189941

curl -X PUT "http://169.254.169.254/latest/api/token" -H "Expect: 100-continue" -H "X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 30" -v
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.61.1
> Accept: */*
> Expect: 100-continue
> X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 30
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 30
< Content-Length: 56
< Date: Sat, 10 Jul 2021 11:12:13 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain
< 
* Closing connection 0
xxxxxxxxxxxxxxxxxxxxxxxxx-the-token-xxxxxxxxxxx
<!-- gh-comment-id:877621511 --> @jvassev commented on GitHub (Jul 10, 2021): I can confirm the same with V1.89 on AmazonLinux. I performed the same request with curl to see if it can handle AWS's implementation as doubted in https://github.com/s3fs-fuse/s3fs-fuse/issues/1581#issuecomment-792189941 ``` curl -X PUT "http://169.254.169.254/latest/api/token" -H "Expect: 100-continue" -H "X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 30" -v * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) > PUT /latest/api/token HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.61.1 > Accept: */* > Expect: 100-continue > X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 30 > < HTTP/1.1 100 Continue < HTTP/1.1 200 OK < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 30 < Content-Length: 56 < Date: Sat, 10 Jul 2021 11:12:13 GMT < Server: EC2ws < Connection: close < Content-Type: text/plain < * Closing connection 0 xxxxxxxxxxxxxxxxxxxxxxxxx-the-token-xxxxxxxxxxx ```
Author
Owner

@maresb commented on GitHub (Aug 10, 2021):

I'm seeing this too. Reverting to 1.87 seems to solve the problem for me.

<!-- gh-comment-id:896298667 --> @maresb commented on GitHub (Aug 10, 2021): I'm seeing this too. Reverting to 1.87 seems to solve the problem for me.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#829
No description provided.