[GH-ISSUE #2021] S3FS mount operation hangs on fuse_main()

kerem commented

2026-03-04 01:50:40 +03:00

Owner

Originally created by @lifayt on GitHub (Aug 11, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2021

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

$ s3fs --version
Amazon Simple Storage Service File System V1.87 (commit:194262c) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Also tested with 1.91

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

$ rpm -qi fuse
Name        : fuse
Version     : 2.9.2
Release     : 8.amzn2
Architecture: x86_64
Install Date: Thu 11 Aug 2022 01:31:20 PM EDT
Group       : System Environment/Base
Size        : 222690
License     : GPL+
Signature   : RSA/SHA256, Thu 16 Nov 2017 12:30:16 PM EST, Key ID 11cf1f95c87f5b1a
Source RPM  : fuse-2.9.2-8.amzn2.src.rpm
Build Date  : Thu 19 Oct 2017 03:18:09 PM EDT
Build Host  : build.amazon.com
Relocations : (not relocatable)
Packager    : Amazon Linux
Vendor      : Amazon Linux
URL         : http://fuse.sf.net
Summary     : File System in Userspace (FUSE) utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE userspace tools to
mount a FUSE filesystem.

Kernel information (uname -r)

$ uname -r
4.14.287-215.504.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

s3fs command line used, if applicable

s3fs -f -s <bucket-name> /mnt/s3_test/ -o dbglevel="debug" -d -o curldbg

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

$ s3fs -f -s -d <bucket_name> /mnt/s3_test -o dbglevel="debug"
[CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [INF] 
[CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [INF] to [DBG] 
[INF]     s3fs.cpp:set_mountpoint_attribute(4373): PROC(uid=1007, gid=10) - MountPoint(uid=1007, gid=10, mode=40700)
[DBG] curl.cpp:InitMimeType(695): Try to load mime types from /etc/mime.types file.
[DBG] curl.cpp:InitMimeType(700): The old mime types are cleared to load new mime types.
[INF] curl.cpp:InitMimeType(723): Loaded mime information from /etc/mime.types
[INF] fdcache.cpp:CheckCacheFileStatTopDir(134): The path to cache top dir is empty, thus not need to check permission.

Details about issue

Once the command is executed, it hangs indefinitely. I was able to figure out where the process hangs by compiling s3fs from source, and then inserting my own debug statements. The process currently hangs in the following area:

  // now passing things off to fuse, fuse will finish evaluating the command line args
  fuse_res = fuse_main(custom_args.argc, custom_args.argv, &s3fs_oper, NULL);

fuse_main never returns, and any parts of s3fs.cpp after this function call don't execute. I'm relatively unfamiliar with c++ so at this point I'm sorta running into a brick wall. Any hints would be helpful!

Originally created by @lifayt on GitHub (Aug 11, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2021 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) ```bash $ s3fs --version Amazon Simple Storage Service File System V1.87 (commit:194262c) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` Also tested with 1.91 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ```bash $ rpm -qi fuse Name : fuse Version : 2.9.2 Release : 8.amzn2 Architecture: x86_64 Install Date: Thu 11 Aug 2022 01:31:20 PM EDT Group : System Environment/Base Size : 222690 License : GPL+ Signature : RSA/SHA256, Thu 16 Nov 2017 12:30:16 PM EST, Key ID 11cf1f95c87f5b1a Source RPM : fuse-2.9.2-8.amzn2.src.rpm Build Date : Thu 19 Oct 2017 03:18:09 PM EDT Build Host : build.amazon.com Relocations : (not relocatable) Packager : Amazon Linux Vendor : Amazon Linux URL : http://fuse.sf.net Summary : File System in Userspace (FUSE) utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE userspace tools to mount a FUSE filesystem. ``` #### Kernel information (uname -r) ``` $ uname -r 4.14.287-215.504.amzn2.x86_64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` $ cat /etc/os-release NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" ``` #### s3fs command line used, if applicable ``` s3fs -f -s <bucket-name> /mnt/s3_test/ -o dbglevel="debug" -d -o curldbg ``` #### /etc/fstab entry, if applicable #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` $ s3fs -f -s -d <bucket_name> /mnt/s3_test -o dbglevel="debug" [CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [INF] [CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [INF] to [DBG] [INF] s3fs.cpp:set_mountpoint_attribute(4373): PROC(uid=1007, gid=10) - MountPoint(uid=1007, gid=10, mode=40700) [DBG] curl.cpp:InitMimeType(695): Try to load mime types from /etc/mime.types file. [DBG] curl.cpp:InitMimeType(700): The old mime types are cleared to load new mime types. [INF] curl.cpp:InitMimeType(723): Loaded mime information from /etc/mime.types [INF] fdcache.cpp:CheckCacheFileStatTopDir(134): The path to cache top dir is empty, thus not need to check permission. ``` ### Details about issue Once the command is executed, it hangs indefinitely. I was able to figure out where the process hangs by compiling s3fs from source, and then inserting my own debug statements. The process currently hangs in the following area: ``` // now passing things off to fuse, fuse will finish evaluating the command line args fuse_res = fuse_main(custom_args.argc, custom_args.argv, &s3fs_oper, NULL); ``` fuse_main never returns, and any parts of s3fs.cpp after this function call don't execute. I'm relatively unfamiliar with c++ so at this point I'm sorta running into a brick wall. Any hints would be helpful!

kerem

2026-03-04 01:50:40 +03:00

closed this issue
added the
need info
label

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@ggtakec commented on GitHub (Aug 14, 2022):

The fuse_main function never returns while s3fs is up and the process is resident, so it doesn't matter that it appears to block in that function.
Normally after the fuse_main call, the log that the s3fs_init function was called should be output, but it seems to be blocked before that.
It seems to be blocked during libfuse processing.

We may need information such as the state of your libfuse installation to resolve this issue.

And since you are using version v1.87, can you try with v1.91 or master branch code if possible?
(#1966 is v1.91, so if the cause is the same, s3fs will be blocked as well.)

@ggtakec commented on GitHub (Aug 14, 2022): The `fuse_main` function never returns while s3fs is up and the process is resident, so it doesn't matter that it appears to block in that function. Normally after the `fuse_main` call, the log that the `s3fs_init` function was called should be output, but it seems to be blocked before that. It seems to be blocked during libfuse processing. We may need information such as the state of your libfuse installation to resolve this issue. And since you are using version v1.87, can you try with v1.91 or master branch code if possible? (#1966 is v1.91, so if the cause is the same, s3fs will be blocked as well.)

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@lifayt commented on GitHub (Aug 15, 2022):

Hi @ggtakec, thanks for responding. I have also tried this with v1.91, the issue is there as well. I installed backwards from 1.91 to 1.87 to see if the issue could be resolved by installing an older version. In terms of libfuse, the process we follow is to install the fuse libs via sudo yum install fuse-2.9.2-11.amzn2 fuse-devel-2.9.2-11.amzn2 fuse-libs-2.9.2-11.amzn2, though I have tried all three available versions of fuse on AL2 (2.9.2-8, 2.9.2-9, 2.9.2-11).

@lifayt commented on GitHub (Aug 15, 2022): Hi @ggtakec, thanks for responding. I have also tried this with v1.91, the issue is there as well. I installed backwards from 1.91 to 1.87 to see if the issue could be resolved by installing an older version. In terms of libfuse, the process we follow is to install the fuse libs via `sudo yum install fuse-2.9.2-11.amzn2 fuse-devel-2.9.2-11.amzn2 fuse-libs-2.9.2-11.amzn2`, though I have tried all three available versions of fuse on AL2 (2.9.2-8, 2.9.2-9, 2.9.2-11).

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@ggtakec commented on GitHub (Aug 15, 2022):

The same phenomenon occurs in v1.91 and v1.87(and intermediate versions?).
Unfortunately I haven't been able to reproduce this yet.
(I need to know why s3fs_init is not being called...)
It doesn't look like there's anything wrong with your libfuse installation, does it? (though 2.9.2 should be fine)

@ggtakec commented on GitHub (Aug 15, 2022): The same phenomenon occurs in v1.91 and v1.87(and intermediate versions?). Unfortunately I haven't been able to reproduce this yet. (I need to know why s3fs_init is not being called...) It doesn't look like there's anything wrong with your libfuse installation, does it? (though 2.9.2 should be fine)

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@lifayt commented on GitHub (Aug 15, 2022):

Our initial hunch is that this is linked to the most recent release of the
AL2 Amazon Machine Image in some way (the issue seems to have cropped up
when we updated to the latest image). The other issue reporting this also
occurs after the release of the most recent AMI.

I think that AMI is the following AL2 release: 2.0.20220719.0

Whats a way I could help debug why s3fs_init is not being called?

On Mon, Aug 15, 2022 at 08:43 Takeshi Nakatani @.***>
wrote:

The same phenomenon occurs in v1.91 and v1.87(and intermediate versions?).
Unfortunately I haven't been able to reproduce this yet.
(I need to know why s3fs_init is not being called...)
It doesn't look like there's anything wrong with your libfuse
installation, does it? (though 2.9.2 should be fine)

—
Reply to this email directly, view it on GitHub
https://github.com/s3fs-fuse/s3fs-fuse/issues/2021#issuecomment-1215029464,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABQWTKONWAV3GZCTW3BHVTDVZJCPJANCNFSM56JCBZQA
.
You are receiving this because you authored the thread.Message ID:
@.***>

@lifayt commented on GitHub (Aug 15, 2022): Our initial hunch is that this is linked to the most recent release of the AL2 Amazon Machine Image in some way (the issue seems to have cropped up when we updated to the latest image). The other issue reporting this also occurs after the release of the most recent AMI. I think that AMI is the following AL2 release: 2.0.20220719.0 Whats a way I could help debug why s3fs_init is not being called? On Mon, Aug 15, 2022 at 08:43 Takeshi Nakatani ***@***.***> wrote: > The same phenomenon occurs in v1.91 and v1.87(and intermediate versions?). > Unfortunately I haven't been able to reproduce this yet. > (I need to know why s3fs_init is not being called...) > It doesn't look like there's anything wrong with your libfuse > installation, does it? (though 2.9.2 should be fine) > > — > Reply to this email directly, view it on GitHub > <https://github.com/s3fs-fuse/s3fs-fuse/issues/2021#issuecomment-1215029464>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABQWTKONWAV3GZCTW3BHVTDVZJCPJANCNFSM56JCBZQA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@ggtakec commented on GitHub (Aug 16, 2022):

I tried with amzn2-ami-kernel-5.10-hvm-2.0.20211201.0-x86_64-gp2 and fuse-2.9.2-11.amzn2.
In this environment I didn't receive any errors and s3fs worked fine.
There may be other conditions as well.
(If it is before fuse_main is called, it may be locked by s3fs processing, but if fuse_main is called, it is difficult to check the cause without reproducing.)

@ggtakec commented on GitHub (Aug 16, 2022): I tried with `amzn2-ami-kernel-5.10-hvm-2.0.20211201.0-x86_64-gp2` and `fuse-2.9.2-11.amzn2`. In this environment I didn't receive any errors and s3fs worked fine. There may be other conditions as well. (If it is before `fuse_main` is called, it may be locked by `s3fs` processing, but if `fuse_main` is called, it is difficult to check the cause without reproducing.)

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@lifayt commented on GitHub (Aug 16, 2022):

That is a pretty old ami - our suspicion is that this started with: Amazon Linux 2 AMI 2.0.20220606.1 x86_64 HVM gp2 or amzn2-ami-hvm-2.0.20220606.1-x86_64-gp2. Are you able to try that one?

Edit: This is probably the newer version of the one you're using:
amzn2-ami-kernel-5.10-hvm-2.0.20220606.1-x86_64-gp2

@lifayt commented on GitHub (Aug 16, 2022): That is a pretty old ami - our suspicion is that this started with: `Amazon Linux 2 AMI 2.0.20220606.1 x86_64 HVM gp2` or `amzn2-ami-hvm-2.0.20220606.1-x86_64-gp2`. Are you able to try that one? Edit: This is probably the newer version of the one you're using: `amzn2-ami-kernel-5.10-hvm-2.0.20220606.1-x86_64-gp2`

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@ggtakec commented on GitHub (Aug 17, 2022):

I couldn't find amzn2-ami-hvm-2.0.20220606.1-x86_64-gp2, but I found amzn2-ami-hvm-2.0.20220719.0-x86_64-gp2 which seems to be newer and tried on it.
It seems that s3fs worked fine on it.(I installed s3fs v1.91 package without building)
I think that if it is reproduced, we are able to proceed towards solving the problem, but I have not been able to reproduce it yet.

@ggtakec commented on GitHub (Aug 17, 2022): I couldn't find `amzn2-ami-hvm-2.0.20220606.1-x86_64-gp2`, but I found `amzn2-ami-hvm-2.0.20220719.0-x86_64-gp2` which seems to be newer and tried on it. It seems that s3fs worked fine on it.(I installed s3fs v1.91 package without building) I think that if it is reproduced, we are able to proceed towards solving the problem, but I have not been able to reproduce it yet.

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@lifayt commented on GitHub (Sep 6, 2022):

Updating to the latest released Amazon AMI resolved this issue, so it seems this was limited to a small 1 or 2 ami slice of releases.

@lifayt commented on GitHub (Sep 6, 2022): Updating to the latest released Amazon AMI resolved this issue, so it seems this was limited to a small 1 or 2 ami slice of releases.

kerem commented

2026-03-04 01:50:41 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 6, 2022):

@lifayt Thank you for your kind and continued checking and reporting!

@ggtakec commented on GitHub (Sep 6, 2022): @lifayt Thank you for your kind and continued checking and reporting!

kerem referenced this issue

2026-03-04 02:02:30 +03:00

[PR #1165] [CLOSED] Fix use_cache='' handling #1858

Rows
Columns

[GH-ISSUE #2021] S3FS mount operation hangs on fuse_main() #1016

Additional Information

Version of s3fs being used (s3fs --version)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Kernel information (uname -r)

GNU/Linux Distribution, if applicable (cat /etc/os-release)

s3fs command line used, if applicable

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Details about issue