[GH-ISSUE #547] RHEL "Transport endpoint is not connected" unless run in foreground #311

New issue

Closed

opened 2026-03-04 01:44:16 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:44:16 +03:00

Owner

Originally created by @tpyro on GitHub (Mar 21, 2017).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/547

Additional Information

Version of s3fs being used (s3fs --version)
V1.80(commit:d40da2c) with NSS
Version of fuse being used (pkg-config --modversion fuse)
2.9.2
System information (uname -a)
Linux ip-172-31-xx-xxx.us-west-2.compute.internal 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux
Distro (cat /etc/issue)
Red Hat Enterprise Linux Server release 7.3 (Maipo)
\S
Kernel \r on an \m
s3fs command line used (if applicable)

[root@ip-172-31-xx-xxx ~]# s3fs tonybucket1 /mnt/1 -d -o curldbg
[root@ip-172-31-xx-xxx ~]# ls /mnt/1
ls: cannot access /mnt/1: Transport endpoint is not connected
[root@ip-172-31-xx-xxx ~]#

/etc/fstab entry (if applicable):
N/A
s3fs syslog messages (grep s3fs /var/log/syslog, or s3fs outputs)
if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2090]: s3fs.cpp:set_s3fs_log_level(254): change debug level from [CRT] to [INF]
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2090]:    PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: s3fs.cpp:s3fs_init(3375): init v1.80(commit:d40da2c) with NSS
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: check services.
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]:      check a bucket.
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]:      URL is http://s3.amazonaws.com/tonybucket1/
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]:      URL changed is http://tonybucket1.s3.amazonaws.com/
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]:      computing signature [GET] [/] [] []
Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]:      url is http://s3.amazonaws.com

Details about issue

I have seen this segfault as well but it's sporadic:

Mar 20 15:46:12 ip-172-31-46-169 s3fs[9166]: s3fs.cpp:s3fs_init(3375): init v1.80(commit:d40da2c) with NSS
Mar 20 15:46:12 ip-172-31-46-169 kernel: s3fs[9169]: segfault at 7fbee0148000 ip 00007fbee92d4b34 sp 00007fbee5822128 error 4 in libc-2.17.so[7fbee918a000+1b6000]

Debug output when (successfully) using the -f foreground option:

[root@ip-172-31-46-169 ~]# s3fs -f -d tonybucket1 /mnt/1
[CRT] s3fs.cpp:set_s3fs_log_level(254): change debug level from [CRT] to [INF]
[INF]     s3fs.cpp:set_mountpoint_attribute(4196): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[CRT] s3fs.cpp:s3fs_init(3375): init v1.80(commit:d40da2c) with NSS
[INF] s3fs.cpp:s3fs_check_service(3753): check services.
[INF]       curl.cpp:CheckBucket(2883): check a bucket.
[INF]       curl.cpp:prepare_url(4199): URL is http://s3.amazonaws.com/tonybucket1/
[INF]       curl.cpp:prepare_url(4231): URL changed is http://tonybucket1.s3.amazonaws.com/
[INF]       curl.cpp:insertV4Headers(2260): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(100): url is http://s3.amazonaws.com
[INF]       curl.cpp:RequestPerform(1945): HTTP response code 400 was returned, returning EIO.
[ERR] curl.cpp:CheckBucket(2922): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'</Message><Region>us-west-2</Region><RequestId>54E5ACE35092352D</RequestId><HostId>iSQDNSYq+H3JGvJPuwP/rOa7NU7tQUT79nmBGTguT9JXBXIWBM4rW+ClXbCCQMcuA/eaBtTfd1k=</HostId></Error>
[CRT] s3fs.cpp:s3fs_check_service(3774): Could not connect wrong region us-east-1, so retry to connect region us-west-2.
[INF]       curl.cpp:CheckBucket(2883): check a bucket.
[INF]       curl.cpp:prepare_url(4199): URL is http://s3-us-west-2.amazonaws.com/tonybucket1/
[INF]       curl.cpp:prepare_url(4231): URL changed is http://tonybucket1.s3-us-west-2.amazonaws.com/
[INF]       curl.cpp:insertV4Headers(2260): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(100): url is http://s3-us-west-2.amazonaws.com
[INF]       curl.cpp:RequestPerform(1933): HTTP response code 200

mount parameters the same whether I use -f or not:

[root@ip-172-31-xx-xxx ~]# mount | grep s3fs
s3fs on /mnt/1 type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0)

Tried specifying URL, passwd_file, endpoint, retries and allow_other, also setenforce=0 (SELinux permissive mode) to no avail.

Originally created by @tpyro on GitHub (Mar 21, 2017). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/547 #### Additional Information - Version of s3fs being used (s3fs --version) V1.80(commit:d40da2c) with NSS - Version of fuse being used (pkg-config --modversion fuse) 2.9.2 - System information (uname -a) Linux ip-172-31-xx-xxx.us-west-2.compute.internal 3.10.0-514.10.2.el7.x86_64 #1 SMP Mon Feb 20 02:37:52 EST 2017 x86_64 x86_64 x86_64 GNU/Linux - Distro (cat /etc/issue) Red Hat Enterprise Linux Server release 7.3 (Maipo) \S Kernel \r on an \m - s3fs command line used (if applicable) ``` [root@ip-172-31-xx-xxx ~]# s3fs tonybucket1 /mnt/1 -d -o curldbg [root@ip-172-31-xx-xxx ~]# ls /mnt/1 ls: cannot access /mnt/1: Transport endpoint is not connected [root@ip-172-31-xx-xxx ~]# ``` - /etc/fstab entry (if applicable): N/A - s3fs syslog messages (grep s3fs /var/log/syslog, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2090]: s3fs.cpp:set_s3fs_log_level(254): change debug level from [CRT] to [INF] Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2090]: PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: s3fs.cpp:s3fs_init(3375): init v1.80(commit:d40da2c) with NSS Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: check services. Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: check a bucket. Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: URL is http://s3.amazonaws.com/tonybucket1/ Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: URL changed is http://tonybucket1.s3.amazonaws.com/ Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: computing signature [GET] [/] [] [] Mar 20 22:57:52 ip-172-31-xx-xxx s3fs[2091]: url is http://s3.amazonaws.com ``` #### Details about issue I have seen this segfault as well but it's sporadic: ``` Mar 20 15:46:12 ip-172-31-46-169 s3fs[9166]: s3fs.cpp:s3fs_init(3375): init v1.80(commit:d40da2c) with NSS Mar 20 15:46:12 ip-172-31-46-169 kernel: s3fs[9169]: segfault at 7fbee0148000 ip 00007fbee92d4b34 sp 00007fbee5822128 error 4 in libc-2.17.so[7fbee918a000+1b6000] ``` Debug output when (successfully) using the -f foreground option: ``` [root@ip-172-31-46-169 ~]# s3fs -f -d tonybucket1 /mnt/1 [CRT] s3fs.cpp:set_s3fs_log_level(254): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4196): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [CRT] s3fs.cpp:s3fs_init(3375): init v1.80(commit:d40da2c) with NSS [INF] s3fs.cpp:s3fs_check_service(3753): check services. [INF] curl.cpp:CheckBucket(2883): check a bucket. [INF] curl.cpp:prepare_url(4199): URL is http://s3.amazonaws.com/tonybucket1/ [INF] curl.cpp:prepare_url(4231): URL changed is http://tonybucket1.s3.amazonaws.com/ [INF] curl.cpp:insertV4Headers(2260): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(100): url is http://s3.amazonaws.com [INF] curl.cpp:RequestPerform(1945): HTTP response code 400 was returned, returning EIO. [ERR] curl.cpp:CheckBucket(2922): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-west-2'</Message><Region>us-west-2</Region><RequestId>54E5ACE35092352D</RequestId><HostId>iSQDNSYq+H3JGvJPuwP/rOa7NU7tQUT79nmBGTguT9JXBXIWBM4rW+ClXbCCQMcuA/eaBtTfd1k=</HostId></Error> [CRT] s3fs.cpp:s3fs_check_service(3774): Could not connect wrong region us-east-1, so retry to connect region us-west-2. [INF] curl.cpp:CheckBucket(2883): check a bucket. [INF] curl.cpp:prepare_url(4199): URL is http://s3-us-west-2.amazonaws.com/tonybucket1/ [INF] curl.cpp:prepare_url(4231): URL changed is http://tonybucket1.s3-us-west-2.amazonaws.com/ [INF] curl.cpp:insertV4Headers(2260): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(100): url is http://s3-us-west-2.amazonaws.com [INF] curl.cpp:RequestPerform(1933): HTTP response code 200 ``` mount parameters the same whether I use -f or not: ``` [root@ip-172-31-xx-xxx ~]# mount | grep s3fs s3fs on /mnt/1 type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0) ``` Tried specifying URL, passwd_file, endpoint, retries and allow_other, also setenforce=0 (SELinux permissive mode) to no avail.

kerem closed this issue

2026-03-04 01:44:17 +03:00

kerem commented

2026-03-04 01:44:17 +03:00

Author

Owner

@ggtakec commented on GitHub (Apr 2, 2017):

@tpyro Thank you for this report.

It seems to be segfault, then please let me know about the following questions.
Has s3fs process terminated immediately after starting s3fs before executing any command(ls etc)?
Or after executing any command(ls etc)?

If you can, would you run s3fs with "-o dbglevel=info -o curldbg" ?
It puts many logs about s3fs and curl, so please let me know about the logs.

Thanks in advance for your help.

@ggtakec commented on GitHub (Apr 2, 2017): @tpyro Thank you for this report. It seems to be segfault, then please let me know about the following questions. Has s3fs process terminated immediately after starting s3fs before executing any command(ls etc)? Or after executing any command(ls etc)? If you can, would you run s3fs with "-o dbglevel=info -o curldbg" ? It puts many logs about s3fs and curl, so please let me know about the logs. Thanks in advance for your help.

kerem commented

2026-03-04 01:44:17 +03:00

Author

Owner

@gaul commented on GitHub (Mar 17, 2019):

Could you test again with 1.85 which includes f53503438c?

@gaul commented on GitHub (Mar 17, 2019): Could you test again with 1.85 which includes f53503438c28910a36fe2b155d6623c9af41d951?

kerem commented

2026-03-04 01:44:17 +03:00

Author

Owner

@gaul commented on GitHub (Jul 11, 2019):

Please test with 1.85 and reopen if symptoms persist.

@gaul commented on GitHub (Jul 11, 2019): Please test with 1.85 and reopen if symptoms persist.

kerem referenced this issue

2026-03-04 01:54:04 +03:00

[PR #311] [MERGED] Change error log to debug log in s3fs_read() #1457