[GH-ISSUE #1966] S3fs mount freezing, attempting to access mount point freezes shell #994

Open
opened 2026-03-04 01:50:30 +03:00 by kerem · 2 comments
Owner

Originally created by @lozbrown on GitHub (Jun 22, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1966

We're having problems with s3fs mounting, until recently this had been working for a long time so we're not sure what the issue is.

The ec2 has an instance role that can access and write to the s3 as confirmed with the AWS cli

however any attempt to use the mountpoint /loz in the command line freezes the shell, we've not been able to access any data.

image

Shell output from foreground mount:

2022-06-22T11:45:58.062Z [INF] s3fs version 1.91(unknown) : s3fs -f -d -o use_cache=/tmp/ -o storage_class=standard_ia -o endpoint=eu-west-2 -o ensure_diskfree=1000 -o allow_other -o dbglevel=debug -o curldbg -o iam_role=auto -o url=https://s3.eu-west-2.amazonaws.com <REDACTED BUCKET>:/TRTH/ /loz
2022-06-22T11:45:58.064Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF]
2022-06-22T11:45:58.064Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [INF] to [DBG]
2022-06-22T11:45:58.064Z [INF]     s3fs.cpp:set_mountpoint_attribute(3628): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40700)
2022-06-22T11:45:58.064Z [DBG] curl.cpp:InitMimeType(400): Try to load mime types from /etc/mime.types file.
2022-06-22T11:45:58.064Z [DBG] curl.cpp:InitMimeType(405): The old mime types are cleared to load new mime types.
2022-06-22T11:45:58.066Z [INF] curl.cpp:InitMimeType(424): Loaded mime information from /etc/mime.types

mountpoint info ls -ltrh /

drwxrw-rw- 2 root root 6 Jun 22 10:11 loz

process info:
ps -ef | grep s3fs
root 15649 3431 0 12:01 pts/0 00:00:00 s3fs -f -d /TRTH/ /loz -o use_cache=/tmp/ -o storage_class=standard_ia -o endpoint=eu-west-2 -o ensure_diskfree=1000 -o allow_other -o dbglevel=debug -o curldbg -o iam_role=auto -o url=https://s3.eu-west-2.amazonaws.com

Version of s3fs being used (s3fs --version)

V1.91

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

rpm -qi fuse
Name : fuse
Version : 2.9.2
Release : 11.amzn2
Architecture: x86_64
Install Date: Thu 16 Jun 2022 02:00:20 PM UTC
Group : System Environment/Base
Size : 222809
License : GPL+
Signature : RSA/SHA256, Thu 06 Dec 2018 07:31:53 PM UTC, Key ID 11cf1f95c87f5b1a
Source RPM : fuse-2.9.2-11.amzn2.src.rpm
Build Date : Fri 16 Nov 2018 08:35:39 PM UTC
Build Host : build.amazon.com
Relocations : (not relocatable)
Packager : Amazon Linux
Vendor : Amazon Linux
URL : https://github.com/libfuse/libfuse
Summary : File System in Userspace (FUSE) utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE userspace tools to
mount a FUSE filesystem.

Kernel information (uname -r)

4.14.276-211.499.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

s3fs command line used, if applicable

s3fs -f -d :/TRTH/ /loz
-o use_cache="/tmp/"
-o storage_class="standard_ia"
-o endpoint="eu-west-2"
-o ensure_diskfree="1000"
-o allow_other
-o dbglevel="debug"
-o curldbg
-o iam_role=auto
-o url="https://s3.eu-west-2.amazonaws.com"

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

Originally created by @lozbrown on GitHub (Jun 22, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1966 We're having problems with s3fs mounting, until recently this had been working for a long time so we're not sure what the issue is. The ec2 has an instance role that can access and write to the s3 as confirmed with the AWS cli however any attempt to use the mountpoint /loz in the command line freezes the shell, we've not been able to access any data. ![image](https://user-images.githubusercontent.com/9961593/175022684-5a991b97-ca15-4276-8e88-3b34fcf7ee25.png) **Shell output from foreground mount:** ``` 2022-06-22T11:45:58.062Z [INF] s3fs version 1.91(unknown) : s3fs -f -d -o use_cache=/tmp/ -o storage_class=standard_ia -o endpoint=eu-west-2 -o ensure_diskfree=1000 -o allow_other -o dbglevel=debug -o curldbg -o iam_role=auto -o url=https://s3.eu-west-2.amazonaws.com <REDACTED BUCKET>:/TRTH/ /loz 2022-06-22T11:45:58.064Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [CRT] to [INF] 2022-06-22T11:45:58.064Z [CRT] s3fs_logger.cpp:LowSetLogLevel(240): change debug level from [INF] to [DBG] 2022-06-22T11:45:58.064Z [INF] s3fs.cpp:set_mountpoint_attribute(3628): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40700) 2022-06-22T11:45:58.064Z [DBG] curl.cpp:InitMimeType(400): Try to load mime types from /etc/mime.types file. 2022-06-22T11:45:58.064Z [DBG] curl.cpp:InitMimeType(405): The old mime types are cleared to load new mime types. 2022-06-22T11:45:58.066Z [INF] curl.cpp:InitMimeType(424): Loaded mime information from /etc/mime.types ``` **mountpoint info ls -ltrh /** drwxrw-rw- 2 root root 6 Jun 22 10:11 loz **process info:** ps -ef | grep s3fs root 15649 3431 0 12:01 pts/0 00:00:00 s3fs -f -d <REDACTED BUCKET> /TRTH/ /loz -o use_cache=/tmp/ -o storage_class=standard_ia -o endpoint=eu-west-2 -o ensure_diskfree=1000 -o allow_other -o dbglevel=debug -o curldbg -o iam_role=auto -o url=https://s3.eu-west-2.amazonaws.com #### Version of s3fs being used (s3fs --version) V1.91 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) rpm -qi fuse Name : fuse Version : 2.9.2 Release : 11.amzn2 Architecture: x86_64 Install Date: Thu 16 Jun 2022 02:00:20 PM UTC Group : System Environment/Base Size : 222809 License : GPL+ Signature : RSA/SHA256, Thu 06 Dec 2018 07:31:53 PM UTC, Key ID 11cf1f95c87f5b1a Source RPM : fuse-2.9.2-11.amzn2.src.rpm Build Date : Fri 16 Nov 2018 08:35:39 PM UTC Build Host : build.amazon.com Relocations : (not relocatable) Packager : Amazon Linux Vendor : Amazon Linux URL : https://github.com/libfuse/libfuse Summary : File System in Userspace (FUSE) utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE userspace tools to mount a FUSE filesystem. #### Kernel information (uname -r) 4.14.276-211.499.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" #### s3fs command line used, if applicable s3fs -f -d <REDACTEDBUCKET>:/TRTH/ /loz \ -o use_cache="/tmp/" \ -o storage_class="standard_ia" \ -o endpoint="eu-west-2" \ -o ensure_diskfree="1000" \ -o allow_other \ -o dbglevel="debug" \ -o curldbg \ -o iam_role=auto \ -o url="https://s3.eu-west-2.amazonaws.com" #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue
Author
Owner

@lifayt commented on GitHub (Aug 12, 2022):

I don't have a solution, but this looks almost exactly like the issue I just opened: #2021. Were you able to find a solution independent of this issue?

<!-- gh-comment-id:1213224432 --> @lifayt commented on GitHub (Aug 12, 2022): I don't have a solution, but this looks almost exactly like the issue I just opened: #2021. Were you able to find a solution independent of this issue?
Author
Owner

@ggtakec commented on GitHub (Aug 14, 2022):

Sorry for my late reply.
From the logs it looks like s3fs is blocked after calling libfuse main function.
Normally, the following line should be output to the log immediately after the attached log, but it is not there.

2022-xx-xxTxx:xx:xx.xxxZ [INF] s3fs.cpp:s3fs_init(3393): init v1.91(commit:49d92c7) with OpenSSL

To resolve this issue, more information, such as the state of libfuse, would be required.
(I think #2021 may be the same cause.)

<!-- gh-comment-id:1214289839 --> @ggtakec commented on GitHub (Aug 14, 2022): Sorry for my late reply. From the logs it looks like s3fs is blocked after calling libfuse main function. Normally, the following line should be output to the log immediately after the attached log, but it is not there. ``` 2022-xx-xxTxx:xx:xx.xxxZ [INF] s3fs.cpp:s3fs_init(3393): init v1.91(commit:49d92c7) with OpenSSL ``` To resolve this issue, more information, such as the state of libfuse, would be required. (I think #2021 may be the same cause.)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#994
No description provided.