starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2457] Getting Frequent dismount issue of OCI object Storage #1210

New issue
Open
opened 2026-03-04 01:52:14 +03:00 by kerem · 9 comments
kerem commented 2026-03-04 01:52:14 +03:00
Owner
Copy link

Originally created by @kparmar-winfo on GitHub (May 23, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2457

Additional Information

Version of s3fs being used (s3fs --version)

V1.93 (commit:unknown) with Open SSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

Name : fuse
Version : 2.9.4
Release : 1.0.9.el7

Kernel information (uname -r)

5.4.17-2136.321.4.el7uek.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://github.com/oracle/oracle-linux"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9

How to run s3fs, if applicable

[] command line
[] /etc/fstab

# MOUNT OBJECT STORE BUCKET AS DISK
s3fs#obj-demo-standard /objstore/demo fuse _netdev,allow_other,use_path_request_style,nomultipart,umask=0000,uid=0,gid=54322,endpoint=uk-london-1,url=https://<NAMESPACE>.compat.objectstorage.uk-london-1.oraclecloud.com/ 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Details about issue

Currently we are using Oracle Linux 7 system in which we have mounted an OCI object storage using s3fs-fuse utility. But since few months we are facing an issue as mountpoint dismounted frequetly with segement fault error.

Please check the below log, generated during dismount of the object storage in /var/log/messages file.

May 19 10:36:03 test01 kernel: s3fs[19840]: segfault at 0 ip 000055a7e80b01ed sp 00007efd2e7fb8c0 error 4 in s3fs[55a7e80a0000+9d000]
May 19 10:36:03 test01 kernel: Code: 00 00 00 48 89 44 24 38 31 c0 49 8d 44 24 18 48 89 44 24 20 31 c0 e8 b2 35 01 00 48 8d 3d 73 ce 06 00 b9 02 00 00 00 48 89 ee a6 74 23 48 8d 3d cc 4e 07 00 b9 02 00 00 00 48 89 ee f3 a6 74
May 19 10:36:03 test01 abrt-hook-ccpp: Process 19698 (s3fs) of user 0 killed by SIGSEGV - dumping core
May 19 10:36:07 test01 abrtd: Size of '/var/spool/abrt' >= 5000 MB (MaxCrashReportsSize), deleting old directory 'ccpp-2024-05-16-10:31:44-18344'

We have also tried to update the s3fs-fuse to it's latest version, though we falll into the same place.

Thank you,
Karan Parmar.

Originally created by @kparmar-winfo on GitHub (May 23, 2024). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2457 <!-- -------------------------------------------------------------------------- The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD. --------------------------------------------------------------------------- --> ### Additional Information #### Version of s3fs being used (`s3fs --version`) <!-- example: V1.91 (commit:b19262a) --> V1.93 (commit:unknown) with Open SSL #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) <!-- example: 2.9.2 --> Name : fuse Version : 2.9.4 Release : 1.0.9.el7 #### Kernel information (`uname -r`) <!-- example: 5.10.96-90.460.amzn2.x86_64 --> 5.4.17-2136.321.4.el7uek.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) <!-- command result --> NAME="Oracle Linux Server" VERSION="7.9" ID="ol" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.9" PRETTY_NAME="Oracle Linux Server 7.9" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:oracle:linux:7:9:server" HOME_URL="https://linux.oracle.com/" BUG_REPORT_URL="https://github.com/oracle/oracle-linux" ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7" ORACLE_BUGZILLA_PRODUCT_VERSION=7.9 ORACLE_SUPPORT_PRODUCT="Oracle Linux" ORACLE_SUPPORT_PRODUCT_VERSION=7.9 #### How to run s3fs, if applicable <!-- Describe the s3fs "command line" or "/etc/fstab" entry used. --> [] command line [] /etc/fstab <!-- Executed command line or /etc/fastab entry --> ``` # MOUNT OBJECT STORE BUCKET AS DISK s3fs#obj-demo-standard /objstore/demo fuse _netdev,allow_other,use_path_request_style,nomultipart,umask=0000,uid=0,gid=54322,endpoint=uk-london-1,url=https://<NAMESPACE>.compat.objectstorage.uk-london-1.oraclecloud.com/ 0 0 ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) <!-- if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages. --> ``` ``` ### Details about issue <!-- Please describe the content of the issue in detail. --> Currently we are using Oracle Linux 7 system in which we have mounted an OCI object storage using s3fs-fuse utility. But since few months we are facing an issue as mountpoint dismounted frequetly with segement fault error. Please check the below log, generated during dismount of the object storage in /var/log/messages file. May 19 10:36:03 test01 kernel: s3fs[19840]: segfault at 0 ip 000055a7e80b01ed sp 00007efd2e7fb8c0 error 4 in s3fs[55a7e80a0000+9d000] May 19 10:36:03 test01 kernel: Code: 00 00 00 48 89 44 24 38 31 c0 49 8d 44 24 18 48 89 44 24 20 31 c0 e8 b2 35 01 00 48 8d 3d 73 ce 06 00 b9 02 00 00 00 48 89 ee <f3> a6 74 23 48 8d 3d cc 4e 07 00 b9 02 00 00 00 48 89 ee f3 a6 74 May 19 10:36:03 test01 abrt-hook-ccpp: Process 19698 (s3fs) of user 0 killed by SIGSEGV - dumping core May 19 10:36:07 test01 abrtd: Size of '/var/spool/abrt' >= 5000 MB (MaxCrashReportsSize), deleting old directory 'ccpp-2024-05-16-10:31:44-18344' We have also tried to update the s3fs-fuse to it's latest version, though we falll into the same place. Thank you, Karan Parmar.
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@kparmar-winfo commented on GitHub (Jun 13, 2024):

We are frequently getting an issue. Can anyone respond to this issue please.

<!-- gh-comment-id:2165596448 --> @kparmar-winfo commented on GitHub (Jun 13, 2024): We are frequently getting an issue. Can anyone respond to this issue please.
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jun 14, 2024):

Please attach gdb to the s3fs process. When the segfault occurs, use thread apply all bt to get the backtrace.

<!-- gh-comment-id:2167332470 --> @gaul commented on GitHub (Jun 14, 2024): Please attach `gdb` to the s3fs process. When the segfault occurs, use `thread apply all bt` to get the backtrace.
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jun 14, 2024):

If you can tolerate the slower performance, valgrind will give an even better diagnostic.

<!-- gh-comment-id:2167333265 --> @gaul commented on GitHub (Jun 14, 2024): If you can tolerate the slower performance, `valgrind` will give an even better diagnostic.
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@kparmar-winfo commented on GitHub (Jun 17, 2024):

s3fs_gdb_backtrace.txt

Hi Gaul,

Have attached s3fs backtrace. Can you plerase check the output once.
Moreover I will try to diagnose process using valgrind and attach that as well later.

Also, I have checked /var/log/messages file in which I found below message. Not sure whether it is showing any error or not.

Jun 17 10:59:55 vm01 s3fs[27838]: Loaded mime information from /etc/mime.types
Jun 17 10:59:55 vm01 s3fs[27851]: init v1.94(commit:unknown) with OpenSSL, credential-library(built-in)
Jun 17 11:00:01 vm01 syst

<!-- gh-comment-id:2172925700 --> @kparmar-winfo commented on GitHub (Jun 17, 2024): [s3fs_gdb_backtrace.txt](https://github.com/user-attachments/files/15866005/s3fs_gdb_backtrace.txt) Hi Gaul, Have attached s3fs backtrace. Can you plerase check the output once. Moreover I will try to diagnose process using valgrind and attach that as well later. Also, I have checked /var/log/messages file in which I found below message. Not sure whether it is showing any error or not. Jun 17 10:59:55 vm01 s3fs[27838]: Loaded mime information from /etc/mime.types Jun 17 10:59:55 vm01 s3fs[27851]: init v1.94(commit:unknown) with OpenSSL, credential-library(built-in) Jun 17 11:00:01 vm01 syst
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@kparmar-winfo commented on GitHub (Jun 17, 2024):

valgrind-out1.txt

Hi Gaul,

We have ran the valgrind for the process of s3fs and attaching the same log here. Can you please check the log once.

<!-- gh-comment-id:2173308653 --> @kparmar-winfo commented on GitHub (Jun 17, 2024): [valgrind-out1.txt](https://github.com/user-attachments/files/15871068/valgrind-out1.txt) Hi Gaul, We have ran the valgrind for the process of s3fs and attaching the same log here. Can you please check the log once.
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jun 17, 2024):

gdb doesn't show a segfault -- are you sure you attached to the process that is failing? The valgrind output is more interesting:

==27851== Invalid read of size 1
==27851==    at 0x11A21C: ??? (in /usr/bin/s3fs)
==27851==    by 0x123C45: ??? (in /usr/bin/s3fs)
==27851==    by 0x4E4D8A6: ??? (in /usr/lib64/libfuse.so.2.9.4)
...
==27851==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

But it appears that s3fs lacks debug symbols so I don't know where the failure is occuring. Which distribution do you have? Can you install debug symbols for s3fs separately? Otherwise you will need to compile s3fs locally to make sure the -g compiler flag is specified so we can make progress.

<!-- gh-comment-id:2174626005 --> @gaul commented on GitHub (Jun 17, 2024): gdb doesn't show a segfault -- are you sure you attached to the process that is failing? The valgrind output is more interesting: ``` ==27851== Invalid read of size 1 ==27851== at 0x11A21C: ??? (in /usr/bin/s3fs) ==27851== by 0x123C45: ??? (in /usr/bin/s3fs) ==27851== by 0x4E4D8A6: ??? (in /usr/lib64/libfuse.so.2.9.4) ... ==27851== Address 0x0 is not stack'd, malloc'd or (recently) free'd ``` But it appears that s3fs lacks debug symbols so I don't know where the failure is occuring. Which distribution do you have? Can you install debug symbols for s3fs separately? Otherwise you will need to compile s3fs locally to make sure the `-g` compiler flag is specified so we can make progress.
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@kparmar-winfo commented on GitHub (Jun 18, 2024):

An error message is reported in dmesg:

[root@vm01~]# dmesg|tail
[14234014.770576] s3fs[30277]: segfault at 0 ip 000055946b24521c sp 00007f8884ff8890 error 4 in s3fs[55946b233000+b8000]
[14234014.770594] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14252238.490976] s3fs[13801]: segfault at 0 ip 000055c1deeb921c sp 00007f4edffbe890 error 4 in s3fs[55c1deea7000+b8000]
[14252238.490983] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14253956.891204] s3fs[22090]: segfault at 0 ip 000055d4af4c821c sp 00007fc43124e890 error 4 in s3fs[55d4af4b6000+b8000]
[14253956.891220] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14266639.509252] s3fs[4421]: segfault at 0 ip 00005596be0fb21c sp 00007f41fa7fb890 error 4 in s3fs[5596be0e9000+b8000]
[14266639.509270] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74
[14337729.456076] s3fs[2395]: segfault at 0 ip 0000561d8d0f521c sp 00007fcd977fd890 error 4 in s3fs[561d8d0e3000+b8000]
[14337729.456083] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74

In addition, I have added -o dbglevel=debug option while mounting s3fs bucket using valgrind. I will send it to you once error occur.

Also, not sure why am I getting commit as unknow:

[root@vm01~]# s3fs --version
Amazon Simple Storage Service File System V1.94 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[root@vm01~]#
<!-- gh-comment-id:2175887504 --> @kparmar-winfo commented on GitHub (Jun 18, 2024): An error message is reported in dmesg: ``` [root@vm01~]# dmesg|tail [14234014.770576] s3fs[30277]: segfault at 0 ip 000055946b24521c sp 00007f8884ff8890 error 4 in s3fs[55946b233000+b8000] [14234014.770594] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74 [14252238.490976] s3fs[13801]: segfault at 0 ip 000055c1deeb921c sp 00007f4edffbe890 error 4 in s3fs[55c1deea7000+b8000] [14252238.490983] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74 [14253956.891204] s3fs[22090]: segfault at 0 ip 000055d4af4c821c sp 00007fc43124e890 error 4 in s3fs[55d4af4b6000+b8000] [14253956.891220] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74 [14266639.509252] s3fs[4421]: segfault at 0 ip 00005596be0fb21c sp 00007f41fa7fb890 error 4 in s3fs[5596be0e9000+b8000] [14266639.509270] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74 [14337729.456076] s3fs[2395]: segfault at 0 ip 0000561d8d0f521c sp 00007fcd977fd890 error 4 in s3fs[561d8d0e3000+b8000] [14337729.456083] Code: 28 00 00 00 48 89 44 24 58 31 c0 48 8d 45 18 48 89 44 24 20 31 c0 e8 b3 99 01 00 48 8d 3d 75 f3 07 00 b9 02 00 00 00 4c 89 e6 <f3> a6 74 26 48 8d 3d 03 76 08 00 b9 02 00 00 00 4c 89 e6 f3 a6 74 ``` In addition, I have added `-o dbglevel=debug` option while mounting s3fs bucket using valgrind. I will send it to you once error occur. Also, not sure why am I getting commit as unknow: ``` [root@vm01~]# s3fs --version Amazon Simple Storage Service File System V1.94 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. [root@vm01~]# ```
kerem commented 2026-03-04 01:52:15 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 23, 2024):

@kparmar-winfo

In addition, I have added -o dbglevel=debug option while mounting s3fs bucket using valgrind. I will send it to you once error occur.

Thanks. I hope the log you get gives some hints.

Also, not sure why am I getting commit as unknow:

I think the reason why unknown is displayed is because the distributed package does not read the git commit hash.
The s3fs distribution package displays this.
(If you built it yourself, you can display the commit hash value if you have the git information.)
This display is misleading, so I will submit a PR to improve it soon.

<!-- gh-comment-id:2184682453 --> @ggtakec commented on GitHub (Jun 23, 2024): @kparmar-winfo > In addition, I have added -o dbglevel=debug option while mounting s3fs bucket using valgrind. I will send it to you once error occur. Thanks. I hope the log you get gives some hints. > Also, not sure why am I getting commit as unknow: I think the reason why `unknown` is displayed is because the distributed package does not read the git commit hash. The s3fs distribution package displays this. _(If you built it yourself, you can display the commit hash value if you have the git information.)_ This display is misleading, so I will submit a PR to improve it soon.
kerem commented 2026-03-04 01:52:16 +03:00
Author
Owner
Copy link

@kparmar-winfo commented on GitHub (Jul 10, 2024):

Hi @ggtakec,

Did we got any progress on the above issue.

Thanks in advance.

<!-- gh-comment-id:2219658743 --> @kparmar-winfo commented on GitHub (Jul 10, 2024): Hi @ggtakec, Did we got any progress on the above issue. Thanks in advance.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1210
No description provided.