mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 05:16:00 +03:00
[GH-ISSUE #1191] s3fs crash during cache read #630
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#630
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Regressor on GitHub (Nov 5, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1191
Version of s3fs being used (s3fs --version)
Amazon Simple Storage Service File System V1.85 (commit:0e815c2) with OpenSSL
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
Version: 2.9.4-1ubuntu3.1
Kernel information (uname -r)
4.4.0-1092-aws
GNU/Linux Distribution, if applicable (cat /etc/os-release)
Description: Ubuntu 16.04.4 LTS
s3fs command line used, if applicable
s3fs media.xxxxxxxx.com /opt/media -o rw,allow_other,nonempty,use_cache=/cache,multireq_max=100,ensure_diskfree=10000,use_path_request_style,endpoint=eu-west-1,url=http://s3-eu-west-1.amazonaws.com,dev,suid
/etc/fstab entry, if applicable
s3fs#media.xxxxxxxx.com /opt/media fuse _netdev,allow_other,nonempty,use_cache=/cache,multireq_max=100,ensure_diskfree=10000,use_path_request_style,endpoint=eu-west-1,url=http://s3-eu-west-1.amazonaws.com 0 0
s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)
Nov 02 22:41:09 ip-xxx-xx-xx-x sudo[7998]: root : TTY=pts/3 ; PWD=/var/crash ; USER=root ; COMMAND=/usr/bin/apport-unpack /var/crash/_usr_bin_s3fs.0.crash s3fscrash
Nov 02 22:45:05 ip-xx-xx-xx-x sudo[8579]: root : TTY=pts/3 ; PWD=/var/crash ; USER=root ; COMMAND=/usr/bin/apport-unpack /var/crash/_usr_bin_s3fs.0.crash s3fscrash
Details about issue
Hi. We have a problem with s3fs. Our webserver distribute media files mounted from s3 bucket. We use caching and sometimes s3fs process crashed and no files can be read or write. We have script checking s3 mount availability that do remounts when problem detected.
When crash happens apport create report. Here is the backtrace:
#0 0x00007fab676b7428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fab676b902a in __GI_abort () at abort.c:89
#2 0x00007fab67f0e84d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007fab67f0c6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007fab67f0c701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007fab67f0c919 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x000000000044e3dc in s3fs_strtoofft (str=, base=base@entry=0) at string_util.cpp:63
#7 0x000000000045d227 in PageList::Serialize (this=this@entry=0x7fab24b510f0, file=..., is_output=is_output@entry=false) at fdcache.cpp:846
#8 0x000000000045f414 in FdEntity::Open (this=this@entry=0x7fab24b51030, pmeta=pmeta@entry=0x7fab0e7eba20, size=size@entry=23483, time=time@entry=1560484571, no_fd_lock_wait=no_fd_lock_wait@entry=false) at fdcache.cpp:1183
#9 0x0000000000461acf in FdManager::Open (this=0x68e620 FdManager::singleton, path=path@entry=0x7fab24b43380 "/cache/d7/f7/d7f7c265c9e6dd0cb8e9b7b6b1bdc8fd.jpg", pmeta=pmeta@entry=0x7fab0e7eba20, size=23483, time=1560484571,
force_tmpfile=force_tmpfile@entry=false, is_create=true, no_fd_lock_wait=false) at fdcache.cpp:2523
#10 0x0000000000416916 in s3fs_open (_path=, fi=0x7fab0e7ebd00) at s3fs.cpp:2191
#11 0x00007fab68c7ba08 in fuse_fs_open () from /lib/x86_64-linux-gnu/libfuse.so.2
#12 0x00007fab68c7bae2 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#13 0x00007fab68c85fdc in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#14 0x00007fab68c85679 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#15 0x00007fab68c81e38 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#16 0x00007fab67a536ba in start_thread (arg=0x7fab0e7ec700) at pthread_create.c:333
#17 0x00007fab6778941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
@Regressor commented on GitHub (Nov 5, 2019):
One more thing. We started with ubuntu packaged 1.76 version and there was no problem, but then we got out of free disk space and got crash. So we updated to 1.85 to use ensure_diskfree option but this version looks like have some bug(s) with cache management.
@ggtakec commented on GitHub (Apr 14, 2020):
@Regressor I’m sorry for my late reply.
This problem seems to have been caused by the internal processing of s3fs (s3fs_strtoofft function) triggered by the same reason as #1245 (file contents are incorrect).
I will upload the patch code soon.
Thanks for your help.
@gaul commented on GitHub (Jun 4, 2020):
@Regressor please test with master and reopen if symptoms persist.