starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1771] s3fs crashes in function get_base_exp #913

New issue
Closed
opened 2026-03-04 01:49:52 +03:00 by kerem · 7 comments
kerem commented 2026-03-04 01:49:52 +03:00
Owner
Copy link

Originally created by @farblos on GitHub (Oct 6, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1771

Version of s3fs being used (s3fs --version)

[root@host ~]# s3fs --version
Amazon Simple Storage Service File System V1.90 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

[root@host ~]# rpm -qi fuse
Name        : fuse
Version     : 2.9.4
Release     : 1.0.9.el7
Architecture: x86_64
Install Date: Thu Apr  1 18:41:34 2021
Group       : System Environment/Base
Size        : 228368
License     : GPL+
Signature   : RSA/SHA256, Tue Feb 23 08:47:27 2021, Key ID 72f97b74ec551f03
Source RPM  : fuse-2.9.4-1.0.9.el7.src.rpm
Build Date  : Tue Feb 23 08:47:03 2021
Build Host  : host-100-100-224-38.blddevtest1iad.osdevelopmeniad.oraclevcn.com
Relocations : (not relocatable)
Vendor      : Oracle America
URL         : http://fuse.sf.net
Summary     : File System in Userspace (FUSE) utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE userspace tools to
mount a FUSE filesystem.

Kernel information (uname -r)

[root@host ~]# uname -r
5.4.17-2102.205.7.3.el7uek.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

[root@host ~]# cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="7.9"
ID="ol"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Oracle Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:9:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.9
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.9

s3fs command line used, if applicable

s3fs -o url=https://url -o passwd_file=$HOME/.passwd-s3fs -o retries=20 -o storage_class=standard_ia -o dbglevel=warn -o use_path_request_style -o endpoint=endpoint -o allow_other -o default_permissions sbp_bucket /mnt

/etc/fstab entry, if applicable

N/A - used s3fs from command line

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Oct 06 05:28:16 host s3fs[26412]: curl.cpp:MapPutErrorResponse(1283): Put request get 200 status response, but it included error body(or NULL). The request failed during copying the object in S3.
Oct 06 07:50:30 host s3fs[26412]: s3fs_xml.cpp:get_base_exp(93): marker_xp->nodesetval is empty.
Oct 06 07:50:30 host s3fs[26412]: s3fs_xml.cpp:get_base_exp(93): marker_xp->nodesetval is empty.
Oct 06 07:50:30 host abrt-hook-ccpp[25990]: Process 26412 (s3fs) of user 0 killed by SIGABRT - dumping core

Details about issue

s3fs crashes in the middle of a rsync-based larger upload (source file system has ~1TB used disk space) with the following backtrace:

  (gdb) backtrace 
  #0  0x00007fd54ef5c387 in raise () from /lib64/libc.so.6
  #1  0x00007fd54ef5da78 in abort () from /lib64/libc.so.6
  #2  0x00007fd54ef9ef67 in __libc_message () from /lib64/libc.so.6
  #3  0x00007fd54efa7329 in _int_free () from /lib64/libc.so.6
  #4  0x000055adc0758484 in get_base_exp(_xmlDoc*, char const*) ()
  #5  0x000055adc075aa4a in append_objects_from_xml(char const*, _xmlDoc*, S3ObjList&) ()
  #6  0x000055adc073321c in list_bucket(char const*, S3ObjList&, char const*, bool) ()
  #7  0x000055adc07387ee in directory_empty(char const*) ()
  #8  0x000055adc074caa8 in s3fs_rename(char const*, char const*) ()
  #9  0x00007fd5507737d7 in fuse_lib_rename () from /lib64/libfuse.so.2
  #10 0x00007fd55077e57f in fuse_ll_process_buf () from /lib64/libfuse.so.2
  #11 0x00007fd55077abfb in fuse_do_work () from /lib64/libfuse.so.2
  #12 0x00007fd54f2fbea5 in start_thread () from /lib64/libpthread.so.0
  #13 0x00007fd54f0249fd in clone () from /lib64/libc.so.6

Out of curiosity I had a look at get_base_exp ... shouldn't there be a test on NULL for this one:

xmlXPathContextPtr ctx = xmlXPathNewContext(doc);

I can provide information from the coredump but the need for screening sensitive information might make this difficult. Reproducing might be impossible.

Originally created by @farblos on GitHub (Oct 6, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1771 #### Version of s3fs being used (s3fs --version) ``` [root@host ~]# s3fs --version Amazon Simple Storage Service File System V1.90 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` [root@host ~]# rpm -qi fuse Name : fuse Version : 2.9.4 Release : 1.0.9.el7 Architecture: x86_64 Install Date: Thu Apr 1 18:41:34 2021 Group : System Environment/Base Size : 228368 License : GPL+ Signature : RSA/SHA256, Tue Feb 23 08:47:27 2021, Key ID 72f97b74ec551f03 Source RPM : fuse-2.9.4-1.0.9.el7.src.rpm Build Date : Tue Feb 23 08:47:03 2021 Build Host : host-100-100-224-38.blddevtest1iad.osdevelopmeniad.oraclevcn.com Relocations : (not relocatable) Vendor : Oracle America URL : http://fuse.sf.net Summary : File System in Userspace (FUSE) utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE userspace tools to mount a FUSE filesystem. ``` #### Kernel information (uname -r) ``` [root@host ~]# uname -r 5.4.17-2102.205.7.3.el7uek.x86_64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` [root@host ~]# cat /etc/os-release NAME="Oracle Linux Server" VERSION="7.9" ID="ol" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.9" PRETTY_NAME="Oracle Linux Server 7.9" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:oracle:linux:7:9:server" HOME_URL="https://linux.oracle.com/" BUG_REPORT_URL="https://bugzilla.oracle.com/" ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7" ORACLE_BUGZILLA_PRODUCT_VERSION=7.9 ORACLE_SUPPORT_PRODUCT="Oracle Linux" ORACLE_SUPPORT_PRODUCT_VERSION=7.9 ``` #### s3fs command line used, if applicable ``` s3fs -o url=https://url -o passwd_file=$HOME/.passwd-s3fs -o retries=20 -o storage_class=standard_ia -o dbglevel=warn -o use_path_request_style -o endpoint=endpoint -o allow_other -o default_permissions sbp_bucket /mnt ``` #### /etc/fstab entry, if applicable N/A - used s3fs from command line #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` Oct 06 05:28:16 host s3fs[26412]: curl.cpp:MapPutErrorResponse(1283): Put request get 200 status response, but it included error body(or NULL). The request failed during copying the object in S3. Oct 06 07:50:30 host s3fs[26412]: s3fs_xml.cpp:get_base_exp(93): marker_xp->nodesetval is empty. Oct 06 07:50:30 host s3fs[26412]: s3fs_xml.cpp:get_base_exp(93): marker_xp->nodesetval is empty. Oct 06 07:50:30 host abrt-hook-ccpp[25990]: Process 26412 (s3fs) of user 0 killed by SIGABRT - dumping core ``` ### Details about issue s3fs crashes in the middle of a rsync-based larger upload (source file system has ~1TB used disk space) with the following backtrace: ``` (gdb) backtrace #0 0x00007fd54ef5c387 in raise () from /lib64/libc.so.6 #1 0x00007fd54ef5da78 in abort () from /lib64/libc.so.6 #2 0x00007fd54ef9ef67 in __libc_message () from /lib64/libc.so.6 #3 0x00007fd54efa7329 in _int_free () from /lib64/libc.so.6 #4 0x000055adc0758484 in get_base_exp(_xmlDoc*, char const*) () #5 0x000055adc075aa4a in append_objects_from_xml(char const*, _xmlDoc*, S3ObjList&) () #6 0x000055adc073321c in list_bucket(char const*, S3ObjList&, char const*, bool) () #7 0x000055adc07387ee in directory_empty(char const*) () #8 0x000055adc074caa8 in s3fs_rename(char const*, char const*) () #9 0x00007fd5507737d7 in fuse_lib_rename () from /lib64/libfuse.so.2 #10 0x00007fd55077e57f in fuse_ll_process_buf () from /lib64/libfuse.so.2 #11 0x00007fd55077abfb in fuse_do_work () from /lib64/libfuse.so.2 #12 0x00007fd54f2fbea5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fd54f0249fd in clone () from /lib64/libc.so.6 ``` Out of curiosity I had a look at `get_base_exp` ... shouldn't there be a test on NULL for this one: ``` xmlXPathContextPtr ctx = xmlXPathNewContext(doc); ``` I can provide information from the coredump but the need for screening sensitive information might make this difficult. Reproducing might be impossible.
kerem closed this issue 2026-03-04 01:49:52 +03:00
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@CarstenGrohmann commented on GitHub (Oct 26, 2021):

Can you share the line number of frame 4 (get_base_exp()), please. Maybe this issue is similar to #1772.

<!-- gh-comment-id:951725671 --> @CarstenGrohmann commented on GitHub (Oct 26, 2021): Can you share the line number of frame 4 (`get_base_exp()`), please. Maybe this issue is similar to #1772.
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@farblos commented on GitHub (Oct 26, 2021):

Will try. IIRC I could not find the debug symbol package corresponding to package s3fs-fuse-1.90-1.el7.x86_64, downloaded from Oracle Linux 7 EPEL Packages.

Any other chances to get these except compiling s3fs myself?

<!-- gh-comment-id:951800277 --> @farblos commented on GitHub (Oct 26, 2021): Will try. IIRC I could not find the debug symbol package corresponding to package `s3fs-fuse-1.90-1.el7.x86_64`, downloaded from `Oracle Linux 7 EPEL Packages`. Any other chances to get these except compiling s3fs myself?
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Oct 26, 2021):

@farblos I'm sorry for my late replying.
I posted PR #1789 to fix this bug.
If you can still try it, please try the code #1789.
Thanks in advance for your kindness.

<!-- gh-comment-id:951853237 --> @ggtakec commented on GitHub (Oct 26, 2021): @farblos I'm sorry for my late replying. I posted PR #1789 to fix this bug. If you can still try it, please try the code #1789. Thanks in advance for your kindness.
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@farblos commented on GitHub (Nov 5, 2021):

@ggtakec I'd like to built v1.90 + your fixes. Since I do not use git on a daily basis, please confirm that the following steps would get me the sources as needed:

git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse
git checkout tags/v1.90
git cherry-pick 78126ae
# fix conflict in s3fs_xml.cpp
git add src/s3fs_xml.cpp
git commit
git cherry-pick 7273d56

Thanks!

<!-- gh-comment-id:962059889 --> @farblos commented on GitHub (Nov 5, 2021): @ggtakec I'd like to built v1.90 + your fixes. Since I do not use git on a daily basis, please confirm that the following steps would get me the sources as needed: ``` git clone https://github.com/s3fs-fuse/s3fs-fuse.git cd s3fs-fuse git checkout tags/v1.90 git cherry-pick 78126ae # fix conflict in s3fs_xml.cpp git add src/s3fs_xml.cpp git commit git cherry-pick 7273d56 ``` Thanks!
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@farblos commented on GitHub (Nov 8, 2021):

At least executing above steps seem to result in a working s3fs executable. Will do some larger scale uploads now using rsync.

<!-- gh-comment-id:963312057 --> @farblos commented on GitHub (Nov 8, 2021): At least executing above steps seem to result in a working `s3fs` executable. Will do some larger scale uploads now using rsync.
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@farblos commented on GitHub (Nov 10, 2021):

Tests so far did not reproduce the problem.

But I observed a slow but steady increase of memory consumption by the s3fs process. Not anything close to an OOM condition, but when s3fs crashed with the stacktrace above it has been running already more than 48 hours of uploads.

IMHO this issue is more of #1483 than anything caused by a race condition. Will try to use valgrind to get more information.

<!-- gh-comment-id:964945211 --> @farblos commented on GitHub (Nov 10, 2021): Tests so far did not reproduce the problem. **But** I observed a slow but steady increase of memory consumption by the s3fs process. Not anything close to an OOM condition, but when s3fs crashed with the stacktrace above it has been running already more than 48 hours of uploads. IMHO this issue is more of #1483 than anything caused by a race condition. Will try to use valgrind to get more information.
kerem commented 2026-03-04 01:49:53 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 26, 2023):

Close this issue. If you still need it, please reopen it.

<!-- gh-comment-id:1484132504 --> @ggtakec commented on GitHub (Mar 26, 2023): Close this issue. If you still need it, please reopen it.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#913
No description provided.