[GH-ISSUE #1429] Segmentation Fault with s3fs 1.87 #756

New issue

Closed

opened 2026-03-04 01:48:30 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:48:30 +03:00

Owner

Originally created by @adwaitsathe on GitHub (Sep 29, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1429

Additional Information

Running s3fs in docker container. The work load is generated by Cassandra stress tool. Results in segmentation fault. Same problem persisted with 1.86 version.

Mount Command:

s3fs vcos-cassandra-1 /var/lib/cassandra -o passwd_file=/etc/cassandra/.vcos-cassandra-1 -o url=http://127.0.0.1:16635 -o use_path_request_style -o kernel_cache -o max_background=1000 -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=5 -o multireq_max=5 -o dbglevel=dbg -o uid=999 -o gid=999 -o allow_other -o mp_umask=002

Thread 159 "s3fs" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fe4337fe700 (LWP 2031)]
PageList::IsModified (this=0x7fe470010b28) at fdcache.cpp:987
987     if(iter->modified){
(gdb) backtrace
#0  PageList::IsModified (this=0x7fe470010b28) at fdcache.cpp:987
#1  FdEntity::IsModified (this=0x7fe470010a60) at fdcache.h:185
#2  FdManager::Open (this=this@entry=0x557da37d9ce0 <FdManager::singleton>, path=path@entry=0x7fe450002c60 "/commitlog/CommitLog-6-1601392740779.log", pmeta=pmeta@entry=0x0, 
    size=size@entry=-1, time=time@entry=-1, force_tmpfile=force_tmpfile@entry=false, is_create=false, no_fd_lock_wait=false) at fdcache.cpp:2943
#3  0x0000557da35a8190 in FdManager::ExistOpen (this=0x557da37d9ce0 <FdManager::singleton>, path=0x7fe450002c60 "/commitlog/CommitLog-6-1601392740779.log", existfd=121, 
    ignore_existfd=false) at fdcache.cpp:2996
#4  0x0000557da35466d0 in s3fs_write (_path=<optimized out>, buf=0x7fe474034870 "䂈\035\026\027\236\257\nӀW\b;ØA\226\027#ra", size=131072, offset=21577728, fi=0x7fe4337fdd10)
    at s3fs.cpp:2232
#5  0x00007fe48ae9c43d in fuse_fs_write_buf () from /lib/x86_64-linux-gnu/libfuse.so.2
#6  0x00007fe48ae9c5b8 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#7  0x00007fe48aea578d in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#8  0x00007fe48aea1e68 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#9  0x00007fe489bcc6db in start_thread (arg=0x7fe4337fe700) at pthread_create.c:463
#10 0x00007fe4898f5a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.87

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

root@localhost:/opt/cassandra# apt-cache policy fuse
fuse:
Installed: 2.9.7-1ubuntu1
Candidate: 2.9.7-1ubuntu1
Version table:
*** 2.9.7-1ubuntu1 500
500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
100 /var/lib/dpkg/status

Kernel information (uname -r)

3.10.0-1062.1.2.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

root@localhost:/opt/cassandra# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(3060): [ent->file=/commitlog/CommitLog-6-1601392740779.log][ent->fd=121]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Dup(1410): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=12]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(1363): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=11]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Write(2475): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][offset=21446656][size=131072]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(1363): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=10]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(1363): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=9]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Dup(1410): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=10]
Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(3060): [ent->file=/commitlog/CommitLog-6-1601392740779.log][ent->fd=121]
Sep 29 17:37:48 localhost s3fs[25]: s3fs.cpp:s3fs_write(2229): [path=/commitlog/CommitLog-6-1601392740779.log][size=131072][offset=21970944][fd=121]

Originally created by @adwaitsathe on GitHub (Sep 29, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1429 ### Additional Information Running s3fs in docker container. The work load is generated by Cassandra stress tool. Results in segmentation fault. Same problem persisted with 1.86 version. Mount Command: > s3fs vcos-cassandra-1 /var/lib/cassandra -o passwd_file=/etc/cassandra/.vcos-cassandra-1 -o url=http://127.0.0.1:16635 -o use_path_request_style -o kernel_cache -o max_background=1000 -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=5 -o multireq_max=5 -o dbglevel=dbg -o uid=999 -o gid=999 -o allow_other -o mp_umask=002 ``` Thread 159 "s3fs" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fe4337fe700 (LWP 2031)] PageList::IsModified (this=0x7fe470010b28) at fdcache.cpp:987 987 if(iter->modified){ (gdb) backtrace #0 PageList::IsModified (this=0x7fe470010b28) at fdcache.cpp:987 #1 FdEntity::IsModified (this=0x7fe470010a60) at fdcache.h:185 #2 FdManager::Open (this=this@entry=0x557da37d9ce0 <FdManager::singleton>, path=path@entry=0x7fe450002c60 "/commitlog/CommitLog-6-1601392740779.log", pmeta=pmeta@entry=0x0, size=size@entry=-1, time=time@entry=-1, force_tmpfile=force_tmpfile@entry=false, is_create=false, no_fd_lock_wait=false) at fdcache.cpp:2943 #3 0x0000557da35a8190 in FdManager::ExistOpen (this=0x557da37d9ce0 <FdManager::singleton>, path=0x7fe450002c60 "/commitlog/CommitLog-6-1601392740779.log", existfd=121, ignore_existfd=false) at fdcache.cpp:2996 #4 0x0000557da35466d0 in s3fs_write (_path=<optimized out>, buf=0x7fe474034870 "䂈\035\026\027\236\257\nӀW\b;ØA\226\027#ra", size=131072, offset=21577728, fi=0x7fe4337fdd10) at s3fs.cpp:2232 #5 0x00007fe48ae9c43d in fuse_fs_write_buf () from /lib/x86_64-linux-gnu/libfuse.so.2 #6 0x00007fe48ae9c5b8 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2 #7 0x00007fe48aea578d in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2 #8 0x00007fe48aea1e68 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2 #9 0x00007fe489bcc6db in start_thread (arg=0x7fe4337fe700) at pthread_create.c:463 #10 0x00007fe4898f5a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 ``` #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.87 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) root@localhost:/opt/cassandra# apt-cache policy fuse fuse: Installed: 2.9.7-1ubuntu1 Candidate: 2.9.7-1ubuntu1 Version table: *** 2.9.7-1ubuntu1 500 500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages 100 /var/lib/dpkg/status #### Kernel information (uname -r) 3.10.0-1062.1.2.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) root@localhost:/opt/cassandra# cat /etc/os-release NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(3060): [ent->file=/commitlog/CommitLog-6-1601392740779.log][ent->fd=121] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Dup(1410): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=12] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(1363): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=11] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Write(2475): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][offset=21446656][size=131072] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(1363): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=10] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(1363): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=9] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Dup(1410): [path=/commitlog/CommitLog-6-1601392740779.log][fd=121][refcnt=10] Sep 29 17:11:54 localhost s3fs[25]: fdcache.cpp:Close(3060): [ent->file=/commitlog/CommitLog-6-1601392740779.log][ent->fd=121] Sep 29 17:37:48 localhost s3fs[25]: s3fs.cpp:s3fs_write(2229): [path=/commitlog/CommitLog-6-1601392740779.log][size=131072][offset=21970944][fd=121] ```

kerem

2026-03-04 01:48:30 +03:00

closed this issue
added the
need info
label

kerem commented

2026-03-04 01:48:31 +03:00

Author

Owner

@gaul commented on GitHub (Sep 29, 2020):

Could you try testing with master? #1363 may have fixed this.

@gaul commented on GitHub (Sep 29, 2020): Could you try testing with master? #1363 may have fixed this.

kerem commented

2026-03-04 01:48:31 +03:00

Author

Owner

@adwaitsathe commented on GitHub (Sep 30, 2020):

The issue seems to have been fixed in master. Could not reproduce it with master.

@adwaitsathe commented on GitHub (Sep 30, 2020): The issue seems to have been fixed in master. Could not reproduce it with master.

kerem referenced this issue

2026-03-04 02:01:23 +03:00

[PR #756] [MERGED] Optimize defaults #1626