[GH-ISSUE #2397] s3fs freezes when running a working command with the -f foreground option #1180

New issue

Open

opened 2026-03-04 01:51:58 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:51:58 +03:00

Owner

Originally created by @GrimzEcho on GitHub (Jan 10, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2397

Details about issue

While troubleshooting an issue with mounting a specific bucket, I attempted to use the -f (foreground) option. This resulted in the terminal running the foreground s3fs process completely freezing. It was not responsive to any ctrl+c or similar commands. Additionally, attempting to perform any disk operations from another terminal (such as ls /tmp or df -h) would hang indefinitely. The only way to recover would be to ps -aux | grep s3fs and kill -9 the stuck s3fs process.

This happens with any s3fs command run in the foreground, even commands that otherwise successfully mount the S3 bucket. Taking a working command (tested several) and just adding the -f option causes the program to freeze

Additional Information

Version of s3fs being used (`s3fs --version`)

s3fs --version

Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt)

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

dpkg -s fuse

Package: fuse
Status: deinstall ok config-files
Priority: optional
Section: utils
Installed-Size: 130
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: amd64
Source: fuse (2.9.9-6)
Version: 2.9.9-6+b1
Config-Version: 2.9.9-6+b1

dpkg -s fuse3

Package: fuse3
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 100
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: amd64
Version: 3.14.0-4
Replaces: fuse
Provides: fuse (= 3.14.0-4)
Depends: libc6 (>= 2.34), libfuse3-3 (= 3.14.0-4), adduser, mount (>= 2.19.1), sed (>= 4)
Breaks: fuse

Kernel information (`uname -r`)

uname -r

6.1.0-17-cloud-amd64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

cat /etc/os-release

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"

Executed command

# works fine
s3fs -o dbglevel=debug -o curldbg -o iam_role=auto <our_bucket>:/<our_prefix>/ /tmp/s3
# no output

ls /tmp/s3
# list of files are displayed
umount /tmp/s3

# freezes -- cannot ctl+c;
# same command, but with -f
s3fs -f -o dbglevel=debug -o curldbg -o iam_role=auto <our_bucket>:/<our_prefix>/ /tmp/s3

2024-01-10T20:51:11.017Z [INF] s3fs version 1.90(unknown) : s3fs -f -o dbglevel=debug -o curldbg -o iam_role=auto <redacted> /tmp/s3
2024-01-10T20:51:11.018Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [DBG] 
2024-01-10T20:51:11.018Z [INF]     s3fs.cpp:set_mountpoint_attribute(4093): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40755)
2024-01-10T20:51:11.018Z [DBG] curl.cpp:InitMimeType(410): Try to load mime types from /etc/mime.types file.
2024-01-10T20:51:11.018Z [DBG] curl.cpp:InitMimeType(415): The old mime types are cleared to load new mime types.
2024-01-10T20:51:11.020Z [INF] curl.cpp:InitMimeType(434): Loaded mime information from /etc/mime.types
2024-01-10T20:51:11.020Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission.
2024-01-10T20:51:11.023Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:unknown) with GnuTLS(gcrypt)
2024-01-10T20:51:11.023Z [INF]       curl.cpp:LoadIAMRoleFromMetaData(2919): Get IAM Role name
2024-01-10T20:51:11.023Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2024-01-10T20:51:11.023Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2024-01-10T20:51:11.023Z [CURL DBG] *   Trying 169.254.169.254:80...
2024-01-10T20:51:11.023Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2024-01-10T20:51:11.023Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
2024-01-10T20:51:11.023Z [CURL DBG] > Host: 169.254.169.254
2024-01-10T20:51:11.023Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-01-10T20:51:11.023Z [CURL DBG] > Accept: */*
2024-01-10T20:51:11.023Z [CURL DBG] > 
2024-01-10T20:51:11.024Z [CURL DBG] < HTTP/1.1 200 OK
2024-01-10T20:51:11.024Z [CURL DBG] < Content-Type: text/plain
2024-01-10T20:51:11.024Z [CURL DBG] < Accept-Ranges: none
2024-01-10T20:51:11.024Z [CURL DBG] < Last-Modified: Wed, 10 Jan 2024 20:32:46 GMT
2024-01-10T20:51:11.024Z [CURL DBG] < Content-Length: 19
2024-01-10T20:51:11.024Z [CURL DBG] < Date: Wed, 10 Jan 2024 20:51:11 GMT
2024-01-10T20:51:11.024Z [CURL DBG] < Server: EC2ws
2024-01-10T20:51:11.024Z [CURL DBG] < Connection: close
2024-01-10T20:51:11.024Z [CURL DBG] < 
2024-01-10T20:51:11.024Z [CURL DBG] * Closing connection 0
2024-01-10T20:51:11.024Z [INF]       curl.cpp:RequestPerform(2316): HTTP response code 200
2024-01-10T20:51:11.024Z [INF]       curl.cpp:SetIAMRoleFromMetaData(1802): IAM role name response = "<redacted_valid_role_name>"
2024-01-10T20:51:11.024Z [INF] s3fs.cpp:s3fs_init(3399): loaded IAM role name = <redacted_valid_role_name>
2024-01-10T20:51:11.024Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2024-01-10T20:51:11.024Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2024-01-10T20:51:11.024Z [INF] s3fs.cpp:s3fs_check_service(3516): check services.
2024-01-10T20:51:11.024Z [INF] curl.cpp:CheckIAMCredentialUpdate(1770): IAM Access Token refreshing...
2024-01-10T20:51:11.024Z [INF]       curl.cpp:GetIAMCredentials(2822): [IAM role=<redacted_valid_role_name>]
2024-01-10T20:51:11.024Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30
2024-01-10T20:51:11.024Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/api/token
2024-01-10T20:51:11.024Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache
2024-01-10T20:51:11.024Z [CURL DBG] *   Trying 169.254.169.254:80...
2024-01-10T20:51:11.024Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1)
2024-01-10T20:51:11.024Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2024-01-10T20:51:11.024Z [CURL DBG] > Host: 169.254.169.254
2024-01-10T20:51:11.024Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2024-01-10T20:51:11.025Z [CURL DBG] > Accept: */*
2024-01-10T20:51:11.025Z [CURL DBG] > Transfer-Encoding: chunked
2024-01-10T20:51:11.025Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2024-01-10T20:51:11.025Z [CURL DBG] > Expect: 100-continue
2024-01-10T20:51:11.025Z [CURL DBG] > 
2024-01-10T20:51:11.025Z [CURL DBG] < HTTP/1.1 100 Continue
Killed

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

journalctl | grep s3fs

# last few lines after the successful ls command
Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: fdcache.cpp:OpenExistFdEntity(628): [path=/<redacted_file_name>][flags=0x0]
Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: fdcache.cpp:Open(526): [path=/<redacted_file_name>[size=-1][time=-1][flags=0x0]
Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: s3fs.cpp:s3fs_getattr(801): [path=/<redacted_file_name>] uid=1000, gid=1000, mode=100640 

# next line is a single time jump to the failed/frozen s3fs
Jan 10 20:51:05 ip-172-31-70-135 s3fs[17565]: destroy

Originally created by @GrimzEcho on GitHub (Jan 10, 2024). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2397  ### Details about issue While troubleshooting an issue with mounting a specific bucket, I attempted to use the `-f (foreground)` option. This resulted in the terminal running the foreground s3fs process completely freezing. It was not responsive to any `ctrl+c` or similar commands. Additionally, attempting to perform any disk operations from another terminal (such as `ls /tmp` or `df -h`) would hang indefinitely. The only way to recover would be to `ps -aux | grep s3fs` and `kill -9 ` the stuck s3fs process. **This happens with any s3fs command run in the foreground, even commands that otherwise successfully mount the S3 bucket. Taking a working command (tested several) and just adding the `-f` option causes the program to freeze** ### Additional Information #### Version of s3fs being used (`s3fs --version`) ``` s3fs --version Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt) ``` #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) ``` dpkg -s fuse Package: fuse Status: deinstall ok config-files Priority: optional Section: utils Installed-Size: 130 Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Architecture: amd64 Source: fuse (2.9.9-6) Version: 2.9.9-6+b1 Config-Version: 2.9.9-6+b1 ``` ``` dpkg -s fuse3 Package: fuse3 Status: install ok installed Priority: optional Section: utils Installed-Size: 100 Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Architecture: amd64 Version: 3.14.0-4 Replaces: fuse Provides: fuse (= 3.14.0-4) Depends: libc6 (>= 2.34), libfuse3-3 (= 3.14.0-4), adduser, mount (>= 2.19.1), sed (>= 4) Breaks: fuse ``` #### Kernel information (`uname -r`) ``` uname -r 6.1.0-17-cloud-amd64 ``` #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) ``` cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" ``` #### Executed command  ``` # works fine s3fs -o dbglevel=debug -o curldbg -o iam_role=auto <our_bucket>:/<our_prefix>/ /tmp/s3 # no output ls /tmp/s3 # list of files are displayed umount /tmp/s3 ``` ``` # freezes -- cannot ctl+c; # same command, but with -f s3fs -f -o dbglevel=debug -o curldbg -o iam_role=auto <our_bucket>:/<our_prefix>/ /tmp/s3 2024-01-10T20:51:11.017Z [INF] s3fs version 1.90(unknown) : s3fs -f -o dbglevel=debug -o curldbg -o iam_role=auto <redacted> /tmp/s3 2024-01-10T20:51:11.018Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [DBG] 2024-01-10T20:51:11.018Z [INF] s3fs.cpp:set_mountpoint_attribute(4093): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40755) 2024-01-10T20:51:11.018Z [DBG] curl.cpp:InitMimeType(410): Try to load mime types from /etc/mime.types file. 2024-01-10T20:51:11.018Z [DBG] curl.cpp:InitMimeType(415): The old mime types are cleared to load new mime types. 2024-01-10T20:51:11.020Z [INF] curl.cpp:InitMimeType(434): Loaded mime information from /etc/mime.types 2024-01-10T20:51:11.020Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission. 2024-01-10T20:51:11.023Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:unknown) with GnuTLS(gcrypt) 2024-01-10T20:51:11.023Z [INF] curl.cpp:LoadIAMRoleFromMetaData(2919): Get IAM Role name 2024-01-10T20:51:11.023Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31 2024-01-10T20:51:11.023Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/ 2024-01-10T20:51:11.023Z [CURL DBG] * Trying 169.254.169.254:80... 2024-01-10T20:51:11.023Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2024-01-10T20:51:11.023Z [CURL DBG] > GET /latest/meta-data/iam/security-credentials/ HTTP/1.1 2024-01-10T20:51:11.023Z [CURL DBG] > Host: 169.254.169.254 2024-01-10T20:51:11.023Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt)) 2024-01-10T20:51:11.023Z [CURL DBG] > Accept: */* 2024-01-10T20:51:11.023Z [CURL DBG] > 2024-01-10T20:51:11.024Z [CURL DBG] < HTTP/1.1 200 OK 2024-01-10T20:51:11.024Z [CURL DBG] < Content-Type: text/plain 2024-01-10T20:51:11.024Z [CURL DBG] < Accept-Ranges: none 2024-01-10T20:51:11.024Z [CURL DBG] < Last-Modified: Wed, 10 Jan 2024 20:32:46 GMT 2024-01-10T20:51:11.024Z [CURL DBG] < Content-Length: 19 2024-01-10T20:51:11.024Z [CURL DBG] < Date: Wed, 10 Jan 2024 20:51:11 GMT 2024-01-10T20:51:11.024Z [CURL DBG] < Server: EC2ws 2024-01-10T20:51:11.024Z [CURL DBG] < Connection: close 2024-01-10T20:51:11.024Z [CURL DBG] < 2024-01-10T20:51:11.024Z [CURL DBG] * Closing connection 0 2024-01-10T20:51:11.024Z [INF] curl.cpp:RequestPerform(2316): HTTP response code 200 2024-01-10T20:51:11.024Z [INF] curl.cpp:SetIAMRoleFromMetaData(1802): IAM role name response = "<redacted_valid_role_name>" 2024-01-10T20:51:11.024Z [INF] s3fs.cpp:s3fs_init(3399): loaded IAM role name = <redacted_valid_role_name> 2024-01-10T20:51:11.024Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool 2024-01-10T20:51:11.024Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler 2024-01-10T20:51:11.024Z [INF] s3fs.cpp:s3fs_check_service(3516): check services. 2024-01-10T20:51:11.024Z [INF] curl.cpp:CheckIAMCredentialUpdate(1770): IAM Access Token refreshing... 2024-01-10T20:51:11.024Z [INF] curl.cpp:GetIAMCredentials(2822): [IAM role=<redacted_valid_role_name>] 2024-01-10T20:51:11.024Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30 2024-01-10T20:51:11.024Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/api/token 2024-01-10T20:51:11.024Z [CURL DBG] * Hostname 169.254.169.254 was found in DNS cache 2024-01-10T20:51:11.024Z [CURL DBG] * Trying 169.254.169.254:80... 2024-01-10T20:51:11.024Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#1) 2024-01-10T20:51:11.024Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2024-01-10T20:51:11.024Z [CURL DBG] > Host: 169.254.169.254 2024-01-10T20:51:11.024Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt)) 2024-01-10T20:51:11.025Z [CURL DBG] > Accept: */* 2024-01-10T20:51:11.025Z [CURL DBG] > Transfer-Encoding: chunked 2024-01-10T20:51:11.025Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2024-01-10T20:51:11.025Z [CURL DBG] > Expect: 100-continue 2024-01-10T20:51:11.025Z [CURL DBG] > 2024-01-10T20:51:11.025Z [CURL DBG] < HTTP/1.1 100 Continue Killed ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` journalctl | grep s3fs # last few lines after the successful ls command Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: fdcache.cpp:OpenExistFdEntity(628): [path=/<redacted_file_name>][flags=0x0] Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: fdcache.cpp:Open(526): [path=/<redacted_file_name>[size=-1][time=-1][flags=0x0] Jan 10 20:49:24 ip-172-31-70-135 s3fs[17565]: s3fs.cpp:s3fs_getattr(801): [path=/<redacted_file_name>] uid=1000, gid=1000, mode=100640 # next line is a single time jump to the failed/frozen s3fs Jan 10 20:51:05 ip-172-31-70-135 s3fs[17565]: destroy ```

kerem commented

2026-03-04 01:51:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 12, 2024):

@BrianHVB Are you trying to use fuse3?
s3fs-fuse does not yet support fuse3.(Code modifications will be required to support it.)
Although it may seem to be working normally, problems will probably occur in various places when using fuse3.

@ggtakec commented on GitHub (Feb 12, 2024): @BrianHVB Are you trying to use fuse3? s3fs-fuse does not yet support fuse3.(Code modifications will be required to support it.) Although it may seem to be working normally, problems will probably occur in various places when using fuse3.

kerem commented

2026-03-04 01:51:59 +03:00

Author

Owner

@theeternalrat commented on GitHub (Jul 14, 2025):

I have a similar setup, Debian 12, s3fs v1.90, no fuse3 installed.

It appears that curl is not communicating properly with IMDSv2:

2025-07-14T22:45:36.823Z [INF]       curl.cpp:GetIAMCredentials(2822): [IAM role=xxx]
2025-07-14T22:45:36.823Z [CURL DBG] *   Trying 169.254.169.254:80...
2025-07-14T22:45:36.824Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
2025-07-14T22:45:36.824Z [CURL DBG] > PUT /latest/api/token HTTP/1.1
2025-07-14T22:45:36.824Z [CURL DBG] > Host: 169.254.169.254
2025-07-14T22:45:36.824Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt))
2025-07-14T22:45:36.824Z [CURL DBG] > Accept: */*
2025-07-14T22:45:36.824Z [CURL DBG] > Transfer-Encoding: chunked
2025-07-14T22:45:36.824Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600
2025-07-14T22:45:36.824Z [CURL DBG] > Expect: 100-continue
2025-07-14T22:45:36.824Z [CURL DBG] >
2025-07-14T22:45:36.824Z [CURL DBG] < HTTP/1.1 100 Continue

The application will hang here until forcibly killed. Compared to a working call to the token service:

> Host: 169.254.169.254
> User-Agent: curl/7.88.1
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Content-Length: 56
< Date: Mon, 14 Jul 2025 22:51:15 GMT
< Server: EC2ws
< Connection: close
< Content-Type: text/plain

I believe the issue is Transfer-Encoding: chunked, but I am not familiar enough to say definitively.
After digging around in the issues more, I found #1581 which has a fix. The next problem being that updated versions of s3fs isn't available in Debian 12.

@theeternalrat commented on GitHub (Jul 14, 2025): I have a similar setup, Debian 12, s3fs v1.90, no fuse3 installed. It appears that curl is not communicating properly with IMDSv2: ```2025-07-14T22:45:36.823Z [INF] curl.cpp:CheckIAMCredentialUpdate(1770): IAM Access Token refreshing... 2025-07-14T22:45:36.823Z [INF] curl.cpp:GetIAMCredentials(2822): [IAM role=xxx] 2025-07-14T22:45:36.823Z [CURL DBG] * Trying 169.254.169.254:80... 2025-07-14T22:45:36.824Z [CURL DBG] * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0) 2025-07-14T22:45:36.824Z [CURL DBG] > PUT /latest/api/token HTTP/1.1 2025-07-14T22:45:36.824Z [CURL DBG] > Host: 169.254.169.254 2025-07-14T22:45:36.824Z [CURL DBG] > User-Agent: s3fs/1.90 (commit hash unknown; GnuTLS(gcrypt)) 2025-07-14T22:45:36.824Z [CURL DBG] > Accept: */* 2025-07-14T22:45:36.824Z [CURL DBG] > Transfer-Encoding: chunked 2025-07-14T22:45:36.824Z [CURL DBG] > X-aws-ec2-metadata-token-ttl-seconds: 21600 2025-07-14T22:45:36.824Z [CURL DBG] > Expect: 100-continue 2025-07-14T22:45:36.824Z [CURL DBG] > 2025-07-14T22:45:36.824Z [CURL DBG] < HTTP/1.1 100 Continue ``` The application will hang here until forcibly killed. Compared to a working call to the token service: ```> PUT /latest/api/token HTTP/1.1 > Host: 169.254.169.254 > User-Agent: curl/7.88.1 > Accept: */* > X-aws-ec2-metadata-token-ttl-seconds: 21600 > < HTTP/1.1 200 OK < X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600 < Content-Length: 56 < Date: Mon, 14 Jul 2025 22:51:15 GMT < Server: EC2ws < Connection: close < Content-Type: text/plain ``` I believe the issue is `Transfer-Encoding: chunked`, but I am not familiar enough to say definitively. After digging around in the issues more, I found #1581 which has a fix. The next problem being that updated versions of s3fs isn't available in Debian 12.