[GH-ISSUE #1256] s3fs 1.86 constant core dump in CentOS container #673

Closed
opened 2026-03-04 01:47:47 +03:00 by kerem · 5 comments
Owner

Originally created by @elstak on GitHub (Mar 24, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1256

Additional Information

centos:7 container running on top of AWS EKS cluster. This was working perfectly fine using s3fs 1.85 installed from EPEL. Since upgrade to 1.86, I get an error on first access:

ls: reading directory /mybucket: Software caused connection abort

If I enable dbglevel=info,curldbg mount options, problem is gone, at least in principle not having done further stress tests.
I tried a newer 1.86-2 version from EPEL-testing with the same result.

Version of s3fs being used (s3fs --version)

1.86

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2

Kernel information (uname -r)

4.14.152-127.182.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

s3fs command line used, if applicable

n/a

/etc/fstab entry, if applicable

s3fs#mybucket	/mybucket	fuse	_netdev,allow_other,iam_role=myrole,use_sse=1,default_acl=bucket-owner-full-control,url=https://s3.us-east-1.amazonaws.com,endpoint=us-east-1,use_path_request_style,use_cache=/tmp/s3fscache,del_cache	0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

No syslog/journalctl available in the container. Dmesg shows nothing.
Core dump files in root fs. This is the backtrace:

Reading symbols from /usr/bin/s3fs...Reading symbols from /usr/lib/debug/usr/bin/s3fs.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `s3fs mybucket /mybucket -o rw,iam_role=myrole'.
Program terminated with signal 6, Aborted.
#0  0x00007f66b4f4b337 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install s3fs-fuse-1.86-1.el7.x86_64
(gdb) bt
#0  0x00007f66b4f4b337 in raise () from /lib64/libc.so.6
#1  0x00007f66b4f4ca28 in abort () from /lib64/libc.so.6
#2  0x00007f66b4f8de87 in __libc_message () from /lib64/libc.so.6
#3  0x00007f66b4f96679 in _int_free () from /lib64/libc.so.6
#4  0x00007f66b5ad5dbe in std::string::assign(std::string const&) () from /lib64/libstdc++.so.6
#5  0x000055dcd1c0428d in operator= (__str=..., this=0x55dcd1e5e2a0 <S3fsCurl::AWSAccessToken>) at /usr/include/c++/4.8.2/bits/basic_string.h:547
#6  S3fsCurl::SetIAMCredentials (response=<optimized out>) at curl.cpp:1882
#7  0x000055dcd1c048ea in S3fsCurl::GetIAMCredentials (this=this@entry=0x7f66927fbaf0) at curl.cpp:2920
#8  0x000055dcd1c04e17 in S3fsCurl::CheckIAMCredentialUpdate () at curl.cpp:1905
#9  0x000055dcd1c04e5d in S3fsCurl::insertAuthHeaders (this=this@entry=0x7f66ac0de860) at curl.cpp:2812
#10 0x000055dcd1c05b58 in S3fsCurl::RequestPerform (this=this@entry=0x7f66ac0de860, dontAddAuthHeaders=dontAddAuthHeaders@entry=false) at curl.cpp:2392
#11 0x000055dcd1c0e5f0 in S3fsMultiCurl::RequestPerformWrapper (arg=0x7f66ac0de860) at curl.cpp:4479
#12 0x00007f66b52eae65 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f66b501388d in clone () from /lib64/libc.so.6

Details about issue

Originally created by @elstak on GitHub (Mar 24, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1256 ### Additional Information centos:7 container running on top of AWS EKS cluster. This was working perfectly fine using s3fs 1.85 installed from EPEL. Since upgrade to 1.86, I get an error on first access: ``` ls: reading directory /mybucket: Software caused connection abort ``` If I enable dbglevel=info,curldbg mount options, problem is gone, at least in principle not having done further stress tests. I tried a newer 1.86-2 version from EPEL-testing with the same result. #### Version of s3fs being used (s3fs --version) 1.86 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.2 #### Kernel information (uname -r) 4.14.152-127.182.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" #### s3fs command line used, if applicable ``` n/a ``` #### /etc/fstab entry, if applicable ``` s3fs#mybucket /mybucket fuse _netdev,allow_other,iam_role=myrole,use_sse=1,default_acl=bucket-owner-full-control,url=https://s3.us-east-1.amazonaws.com,endpoint=us-east-1,use_path_request_style,use_cache=/tmp/s3fscache,del_cache 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` No syslog/journalctl available in the container. Dmesg shows nothing. Core dump files in root fs. This is the backtrace: Reading symbols from /usr/bin/s3fs...Reading symbols from /usr/lib/debug/usr/bin/s3fs.debug...done. done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `s3fs mybucket /mybucket -o rw,iam_role=myrole'. Program terminated with signal 6, Aborted. #0 0x00007f66b4f4b337 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install s3fs-fuse-1.86-1.el7.x86_64 (gdb) bt #0 0x00007f66b4f4b337 in raise () from /lib64/libc.so.6 #1 0x00007f66b4f4ca28 in abort () from /lib64/libc.so.6 #2 0x00007f66b4f8de87 in __libc_message () from /lib64/libc.so.6 #3 0x00007f66b4f96679 in _int_free () from /lib64/libc.so.6 #4 0x00007f66b5ad5dbe in std::string::assign(std::string const&) () from /lib64/libstdc++.so.6 #5 0x000055dcd1c0428d in operator= (__str=..., this=0x55dcd1e5e2a0 <S3fsCurl::AWSAccessToken>) at /usr/include/c++/4.8.2/bits/basic_string.h:547 #6 S3fsCurl::SetIAMCredentials (response=<optimized out>) at curl.cpp:1882 #7 0x000055dcd1c048ea in S3fsCurl::GetIAMCredentials (this=this@entry=0x7f66927fbaf0) at curl.cpp:2920 #8 0x000055dcd1c04e17 in S3fsCurl::CheckIAMCredentialUpdate () at curl.cpp:1905 #9 0x000055dcd1c04e5d in S3fsCurl::insertAuthHeaders (this=this@entry=0x7f66ac0de860) at curl.cpp:2812 #10 0x000055dcd1c05b58 in S3fsCurl::RequestPerform (this=this@entry=0x7f66ac0de860, dontAddAuthHeaders=dontAddAuthHeaders@entry=false) at curl.cpp:2392 #11 0x000055dcd1c0e5f0 in S3fsMultiCurl::RequestPerformWrapper (arg=0x7f66ac0de860) at curl.cpp:4479 #12 0x00007f66b52eae65 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f66b501388d in clone () from /lib64/libc.so.6 ``` ### Details about issue
kerem 2026-03-04 01:47:47 +03:00
Author
Owner

@gaul commented on GitHub (Mar 25, 2020):

This looks like memory corruption. Could you run s3fs under valgrind which may reveal the cause? It also might help to install the debug symbols as gdb suggests via: debuginfo-install s3fs-fuse-1.86-1.el7.x86_64

<!-- gh-comment-id:603836137 --> @gaul commented on GitHub (Mar 25, 2020): This looks like memory corruption. Could you run s3fs under valgrind which may reveal the cause? It also might help to install the debug symbols as gdb suggests via: `debuginfo-install s3fs-fuse-1.86-1.el7.x86_64`
Author
Owner

@elstak commented on GitHub (Mar 25, 2020):

This is the full debug info:

# gdb --core=/core.200
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
[New LWP 217]
[New LWP 216]
[New LWP 223]
[New LWP 221]
[New LWP 220]
[New LWP 201]
[New LWP 218]
[New LWP 225]
[New LWP 219]
[New LWP 224]
[New LWP 200]
[New LWP 222]
[New LWP 202]
[New LWP 231]
[New LWP 226]
[New LWP 234]
[New LWP 215]
[New LWP 227]
[New LWP 230]
[New LWP 233]
[New LWP 228]
[New LWP 232]
[New LWP 229]
Reading symbols from /usr/bin/s3fs...Reading symbols from /usr/lib/debug/usr/bin/s3fs.debug...done.
done.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `s3fs mybucket /mybucket -o rw,allow_other,iam_role=myrole'.
Program terminated with signal 6, Aborted.
#0  0x00007f028e63a337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
Missing separate debuginfos, use: debuginfo-install s3fs-fuse-1.86-1.el7.x86_64
(gdb) bt
#0  0x00007f028e63a337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f028e63ba28 in __GI_abort () at abort.c:90
#2  0x00007f028e67ce87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f028e78f3b8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#3  0x00007f028e685679 in malloc_printerr (ar_ptr=0x7f0248000020, ptr=<optimized out>, str=0x7f028e78f4e0 "double free or corruption (!prev)", action=3) at malloc.c:4967
#4  _int_free (av=0x7f0248000020, p=<optimized out>, have_lock=0) at malloc.c:3843
#5  0x00007f028f1c4dbe in _M_dispose (__a=..., this=<optimized out>) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:254
#6  std::string::assign (this=this@entry=0x564e7caac2a0 <S3fsCurl::AWSAccessToken>,
    __str="FwoGZXIvYXdzEJD//////////wEaDFsvUlN/cx0K2Kul0iKtAac3ndZVe/5AS96Jn4dLzxBrbhwpVnRsOy8aazbYBp7FacNlZQ0b4vJa8FufZadNu6fDTf6m4nTjpJ9gCcrnoG4lnqNOco/FQHnSAzxYvJegE+SsMVyzCZNUrflrdbS9Hj8LtuXsuYBmAtNzzcNjPxej"...)
    at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:250
#7  0x0000564e7c85228d in operator= (__str=..., this=0x564e7caac2a0 <S3fsCurl::AWSAccessToken>) at /usr/include/c++/4.8.2/bits/basic_string.h:547
#8  S3fsCurl::SetIAMCredentials (response=<optimized out>) at curl.cpp:1882
#9  0x0000564e7c8528ea in S3fsCurl::GetIAMCredentials (this=this@entry=0x7f0282ffcaf0) at curl.cpp:2920
#10 0x0000564e7c852e17 in S3fsCurl::CheckIAMCredentialUpdate () at curl.cpp:1905
#11 0x0000564e7c852e5d in S3fsCurl::insertAuthHeaders (this=this@entry=0x7f027c05e4a0) at curl.cpp:2812
#12 0x0000564e7c853b58 in S3fsCurl::RequestPerform (this=this@entry=0x7f027c05e4a0, dontAddAuthHeaders=dontAddAuthHeaders@entry=false) at curl.cpp:2392
#13 0x0000564e7c85c5f0 in S3fsMultiCurl::RequestPerformWrapper (arg=0x7f027c05e4a0) at curl.cpp:4479
#14 0x00007f028e9d9e65 in start_thread (arg=0x7f0282ffd700) at pthread_create.c:307
#15 0x00007f028e70288d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

As usual, running it with valgrind didn't fail:

==394== Memcheck, a memory error detector
==394== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==394== Using Valgrind-3.14.0-353a3587bb-20181007X and LibVEX; rerun with -h for copyright info
==394== Command: s3fs -f mybucket /mybucket-o rw,allow_other,iam_role=myrole,use_sse=1,default_acl=bucket-owner-full-control,url=https://s3.us-east-1.amazonaws.com,endpoint=us-east-1,use_path_request_style,use_cache=/tmp/s3fscache,del_cache,dev,suid
==394== Parent PID: 1
==394==
--394--
--394-- Valgrind options:
--394--    -v
--394--    --tool=memcheck
--394--    --leak-check=full
--394--    --show-leak-kinds=all
--394--    --log-file=/s3fs.valgrind
--394-- Contents of /proc/version:
--394--   Linux version 4.14.152-127.182.amzn2.x86_64 (mockbuild@ip-10-0-1-129) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-6) (GCC)) #1 SMP Thu Nov 14 17:32:43 UTC 2019
--394--
--394-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-avx-avx2-bmi
--394-- Page sizes: currently 4096, max supported 4096
--394-- Valgrind library directory: /usr/lib64/valgrind
--394-- Reading syms from /usr/bin/s3fs
--394--   Considering /usr/lib/debug/.build-id/45/69b66a26f0451df4e4958af1f15ee1dca16d9a.debug ..
--394--   .. build-id is valid
--394-- Warning: cross-CU LIMITATION: some inlined fn names
--394-- might be shown as UnknownInlinedFun
--394-- Reading syms from /usr/lib64/ld-2.17.so
--394--   Considering /usr/lib/debug/.build-id/5c/c1a53b747a7e4d21198723c2b633e54f3c06d9.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/valgrind/memcheck-amd64-linux
--394--    object doesn't have a symbol table
--394--    object doesn't have a dynamic symbol table
--394-- Scheduler: using generic scheduler lock implementation.
--394-- Reading suppressions file: /usr/lib64/valgrind/default.supp
==394== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-394-by-???-on-sdu-rhn-synchronizer
==394== embedded gdbserver: writing to   /tmp/vgdb-pipe-to-vgdb-from-394-by-???-on-sdu-rhn-synchronizer
==394== embedded gdbserver: shared mem   /tmp/vgdb-pipe-shared-mem-vgdb-394-by-???-on-sdu-rhn-synchronizer
==394==
==394== TO CONTROL THIS PROCESS USING vgdb (which you probably
==394== don't want to do, unless you know exactly what you're doing,
==394== or are doing some strange experiment):
==394==   /usr/lib64/valgrind/../../bin/vgdb --pid=394 ...command...
==394==
==394== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==394==   /path/to/gdb s3fs
==394== and then give GDB the following command
==394==   target remote | /usr/lib64/valgrind/../../bin/vgdb --pid=394
==394== --pid is optional if only one valgrind process is running
==394==
--394-- REDIR: 0x4019df0 (ld-linux-x86-64.so.2:strlen) redirected to 0x580c66c5 (???)
--394-- REDIR: 0x4019bc0 (ld-linux-x86-64.so.2:index) redirected to 0x580c66df (???)
--394-- Reading syms from /usr/lib64/valgrind/vgpreload_core-amd64-linux.so
--394-- Reading syms from /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so
==394== WARNING: new redirection conflicts with existing -- ignoring it
--394--     old: 0x04019df0 (strlen              ) R-> (0000.0) 0x580c66c5 ???
--394--     new: 0x04019df0 (strlen              ) R-> (2007.0) 0x04c2d0e0 strlen
--394-- REDIR: 0x4019d70 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4c2e230 (strcmp)
--394-- REDIR: 0x401aa30 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x4c31de0 (mempcpy)
--394-- Reading syms from /usr/lib64/libfuse.so.2.9.2
--394--   Considering /usr/lib/debug/.build-id/01/3cd7d3f5015942bcd20e24a1ba945d1547af5b.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libcurl.so.4.3.0
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libxml2.so.2.9.1
--394--   Considering /usr/lib/debug/.build-id/3a/743aad8d50d50cd1dd10e272d620a1ca7e0f0f.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libcrypto.so.1.0.2k
--394--   Considering /usr/lib/debug/.build-id/4c/f1939f660008cfa869d8364651f31aacd2c1c4.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libstdc++.so.6.0.19
--394--   Considering /usr/lib/debug/.build-id/e2/fb6e9c483d89e8e96d73c7ccf3e3a91e91bb81.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libm-2.17.so
--394--   Considering /usr/lib/debug/.build-id/56/81c054fdabcf789f4dda66e94f1f6ed1747327.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libgcc_s-4.8.5-20150702.so.1
--394--   Considering /usr/lib/debug/.build-id/da/c0179f4555aefec9e97476201802fd20c03ec5.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libpthread-2.17.so
--394--   Considering /usr/lib/debug/.build-id/8b/33f7f8c86f8d544c63c5541a8e42b3ddfef8b1.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libc-2.17.so
--394--   Considering /usr/lib/debug/.build-id/39/8944d32cf16a67af51067a326e6c0cc14f90ed.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libdl-2.17.so
--394--   Considering /usr/lib/debug/.build-id/18/113e6e83d8e981b8e8d808f7f3dbb23f950a1d.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libidn.so.11.6.11
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libssh2.so.1.0.1
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libssl3.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libsmime3.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libnss3.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libnssutil3.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libplds4.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libplc4.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libnspr4.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libgssapi_krb5.so.2.2
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libkrb5.so.3.3
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libk5crypto.so.3.1
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libcom_err.so.2.1
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/liblber-2.4.so.2.10.7
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libldap-2.4.so.2.10.7
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libz.so.1.2.7
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/liblzma.so.5.2.2
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libssl.so.1.0.2k
--394--   Considering /usr/lib/debug/.build-id/3b/305c3ba17fe394862e749763f2956c9c890c2e.debug ..
--394--   .. build-id is valid
--394--   Considering /usr/lib/debug/.build-id/30/e5624c134dbd7e6f484dea8faf929a7e1317a3.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/librt-2.17.so
--394--   Considering /usr/lib/debug/.build-id/47/49697bf078337576c4629f0d30b296a0939779.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libkrb5support.so.0.1
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libkeyutils.so.1.5
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libresolv-2.17.so
--394--   Considering /usr/lib/debug/.build-id/30/09b26b33156eaaf99787aa3da0c6ae99649755.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libsasl2.so.3.0.0
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libselinux.so.1
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libcrypt-2.17.so
--394--   Considering /usr/lib/debug/.build-id/84/467c988f41d853c58353beb247670e15da8bad.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libpcre.so.1.2.0
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libfreebl3.so
--394--    object doesn't have a symbol table
--394-- REDIR: 0x65770e0 (libc.so.6:strcasecmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6573e60 (libc.so.6:strnlen) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65793b0 (libc.so.6:strncasecmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65768c0 (libc.so.6:memset) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6576870 (libc.so.6:memcpy@GLIBC_2.2.5) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6573770 (libc.so.6:strcpy) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6573d30 (libc.so.6:strlen) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x657ba80 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6572020 (libc.so.6:strcat) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65722e0 (libc.so.6:strcmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65fc3b0 (libc.so.6:__memmove_chk) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x658ce90 (libc.so.6:strstr) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6572220 (libc.so.6:index) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6575c80 (libc.so.6:strspn) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65762a0 (libc.so.6:bcmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6573f40 (libc.so.6:strncmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x657d130 (libc.so.6:rawmemchr) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65757d0 (libc.so.6:strncpy) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6575810 (libc.so.6:rindex) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65758f0 (libc.so.6:strpbrk) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6576f40 (libc.so.6:stpcpy) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x65fc1a0 (libc.so.6:__memcpy_chk) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x658d9a0 (libc.so.6:strcasestr) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper)
--394-- REDIR: 0x6573d80 (libc.so.6:__GI_strlen) redirected to 0x4c2d040 (__GI_strlen)
--394-- REDIR: 0x6575850 (libc.so.6:__GI_strrchr) redirected to 0x4c2caa0 (__GI_strrchr)
--394-- REDIR: 0x656ca90 (libc.so.6:malloc) redirected to 0x4c29e1c (malloc)
--394-- REDIR: 0x658c8d0 (libc.so.6:__GI_strstr) redirected to 0x4c32070 (__strstr_sse2)
--394-- REDIR: 0x6575f50 (libc.so.6:memchr) redirected to 0x4c2e2d0 (memchr)
--394-- REDIR: 0x657baf0 (libc.so.6:__GI_memcpy) redirected to 0x4c2eee0 (__GI_memcpy)
--394-- REDIR: 0x6627c70 (libc.so.6:__strstr_sse42) redirected to 0x4c32100 (__strstr_sse42)
--394-- REDIR: 0x656ceb0 (libc.so.6:free) redirected to 0x4c2af16 (free)
--394-- REDIR: 0x6656630 (libc.so.6:__strlen_sse2_pminub) redirected to 0x4c2d020 (strlen)
--394-- REDIR: 0x5b0aeb0 (libstdc++.so.6:operator new(unsigned long)) redirected to 0x4c2a43c (operator new(unsigned long))
--394-- REDIR: 0x663b9f0 (libc.so.6:__memcpy_ssse3_back) redirected to 0x4c2e6e0 (memcpy@@GLIBC_2.14)
--394-- REDIR: 0x65762e0 (libc.so.6:__GI_memcmp) redirected to 0x4c30380 (__GI_memcmp)
--394-- REDIR: 0x6625d40 (libc.so.6:__strcmp_sse42) redirected to 0x4c2e1e0 (__strcmp_sse42)
--394-- REDIR: 0x656d4b0 (libc.so.6:calloc) redirected to 0x4c2bf23 (calloc)
--394-- REDIR: 0xffffffffff600400 (???:???) redirected to 0x580c66b1 (???)
--394-- REDIR: 0x6572260 (libc.so.6:__GI_strchr) redirected to 0x4c2cbd0 (__GI_strchr)
--394-- REDIR: 0x656cf90 (libc.so.6:realloc) redirected to 0x4c2c0f5 (realloc)
--394-- REDIR: 0x6573f80 (libc.so.6:__GI_strncmp) redirected to 0x4c2d870 (__GI_strncmp)
--394-- REDIR: 0x6576f80 (libc.so.6:__GI_stpcpy) redirected to 0x4c30780 (__GI_stpcpy)
--394-- REDIR: 0x6576a90 (libc.so.6:__GI_mempcpy) redirected to 0x4c31b10 (__GI_mempcpy)
--394-- REDIR: 0x6581cc0 (libc.so.6:__strncpy_sse2_unaligned) redirected to 0x4c2d6b0 (__strncpy_sse2_unaligned)
--394-- REDIR: 0x401ab80 (ld-linux-x86-64.so.2:stpcpy) redirected to 0x4c30b00 (stpcpy)
--394-- REDIR: 0x5b0afc0 (libstdc++.so.6:operator new[](unsigned long)) redirected to 0x4c2aae1 (operator new[](unsigned long))
--394-- REDIR: 0x6626af0 (libc.so.6:__strncmp_sse42) redirected to 0x4c2d950 (__strncmp_sse42)
--394-- REDIR: 0x6625c90 (libc.so.6:__strchr_sse42) redirected to 0x4c2cc90 (index)
--394-- REDIR: 0x5b09020 (libstdc++.so.6:operator delete(void*)) redirected to 0x4c2b3c6 (operator delete(void*))
--394-- REDIR: 0x6650f60 (libc.so.6:__memcmp_sse4_1) redirected to 0x4c304c0 (__memcmp_sse4_1)
--394-- REDIR: 0x65766e0 (libc.so.6:__GI_memmove) redirected to 0x4c31170 (__GI_memmove)
--394-- REDIR: 0x6640f90 (libc.so.6:__memmove_ssse3_back) redirected to 0x4c2e390 (memcpy@GLIBC_2.2.5)
--394-- REDIR: 0x6627ad0 (libc.so.6:__strrchr_sse42) redirected to 0x4c2cb30 (__strrchr_sse42)
--394-- REDIR: 0x657d370 (libc.so.6:strchrnul) redirected to 0x4c31900 (strchrnul)
--394-- REDIR: 0x6582ca0 (libc.so.6:__stpcpy_sse2_unaligned) redirected to 0x4c30940 (__stpcpy_sse2_unaligned)
--394-- REDIR: 0x6581690 (libc.so.6:__strcpy_sse2_unaligned) redirected to 0x4c2d100 (strcpy)
--394-- REDIR: 0x663b9e0 (libc.so.6:__memcpy_chk_ssse3_back) redirected to 0x4c31ed0 (__memcpy_chk)
--394-- REDIR: 0x657d160 (libc.so.6:__GI___rawmemchr) redirected to 0x4c31960 (__GI___rawmemchr)
--394-- REDIR: 0x65737b0 (libc.so.6:__GI_strcpy) redirected to 0x4c2d1e0 (__GI_strcpy)
--394-- REDIR: 0x662f750 (libc.so.6:__strncasecmp_avx) redirected to 0x4c2daa0 (strncasecmp)
--394-- REDIR: 0x6576920 (libc.so.6:__GI_memset) redirected to 0x4c30d30 (memset)
--394-- REDIR: 0x662e0e0 (libc.so.6:__strcasecmp_avx) redirected to 0x4c2d9c0 (strcasecmp)
--394-- REDIR: 0x5b09050 (libstdc++.so.6:operator delete[](void*)) redirected to 0x4c2ba38 (operator delete[](void*))
--394-- REDIR: 0x6572320 (libc.so.6:__GI_strcmp) redirected to 0x4c2e140 (__GI_strcmp)
--394-- Reading syms from /usr/lib64/libnss_files-2.17.so
--394--   Considering /usr/lib/debug/.build-id/a1/db0e8103de9f2540788eea6cbce3f639c2b39d.debug ..
--394--   .. build-id is valid
--394-- Reading syms from /usr/lib64/libnss_dns-2.17.so
--394--   Considering /usr/lib/debug/.build-id/e9/451101125dc601a139a6b9c649f348e264aded.debug ..
--394--   .. build-id is valid
--394-- REDIR: 0xffffffffff600000 (???:???) redirected to 0x580c66a7 (???)
--394-- Reading syms from /usr/lib64/libsoftokn3.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libsqlite3.so.0.8.6
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libfreeblpriv3.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libnsssysinit.so
--394--    object doesn't have a symbol table
--394-- Reading syms from /usr/lib64/libnsspem.so
--394--    object doesn't have a symbol table
--394-- REDIR: 0x6580bf0 (libc.so.6:memrchr) redirected to 0x4c2e350 (memrchr)
<!-- gh-comment-id:603876354 --> @elstak commented on GitHub (Mar 25, 2020): This is the full debug info: ``` # gdb --core=/core.200 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. [New LWP 217] [New LWP 216] [New LWP 223] [New LWP 221] [New LWP 220] [New LWP 201] [New LWP 218] [New LWP 225] [New LWP 219] [New LWP 224] [New LWP 200] [New LWP 222] [New LWP 202] [New LWP 231] [New LWP 226] [New LWP 234] [New LWP 215] [New LWP 227] [New LWP 230] [New LWP 233] [New LWP 228] [New LWP 232] [New LWP 229] Reading symbols from /usr/bin/s3fs...Reading symbols from /usr/lib/debug/usr/bin/s3fs.debug...done. done. [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `s3fs mybucket /mybucket -o rw,allow_other,iam_role=myrole'. Program terminated with signal 6, Aborted. #0 0x00007f028e63a337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Missing separate debuginfos, use: debuginfo-install s3fs-fuse-1.86-1.el7.x86_64 (gdb) bt #0 0x00007f028e63a337 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55 #1 0x00007f028e63ba28 in __GI_abort () at abort.c:90 #2 0x00007f028e67ce87 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f028e78f3b8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:196 #3 0x00007f028e685679 in malloc_printerr (ar_ptr=0x7f0248000020, ptr=<optimized out>, str=0x7f028e78f4e0 "double free or corruption (!prev)", action=3) at malloc.c:4967 #4 _int_free (av=0x7f0248000020, p=<optimized out>, have_lock=0) at malloc.c:3843 #5 0x00007f028f1c4dbe in _M_dispose (__a=..., this=<optimized out>) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:254 #6 std::string::assign (this=this@entry=0x564e7caac2a0 <S3fsCurl::AWSAccessToken>, __str="FwoGZXIvYXdzEJD//////////wEaDFsvUlN/cx0K2Kul0iKtAac3ndZVe/5AS96Jn4dLzxBrbhwpVnRsOy8aazbYBp7FacNlZQ0b4vJa8FufZadNu6fDTf6m4nTjpJ9gCcrnoG4lnqNOco/FQHnSAzxYvJegE+SsMVyzCZNUrflrdbS9Hj8LtuXsuYBmAtNzzcNjPxej"...) at /usr/src/debug/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/x86_64-redhat-linux/libstdc++-v3/include/bits/basic_string.tcc:250 #7 0x0000564e7c85228d in operator= (__str=..., this=0x564e7caac2a0 <S3fsCurl::AWSAccessToken>) at /usr/include/c++/4.8.2/bits/basic_string.h:547 #8 S3fsCurl::SetIAMCredentials (response=<optimized out>) at curl.cpp:1882 #9 0x0000564e7c8528ea in S3fsCurl::GetIAMCredentials (this=this@entry=0x7f0282ffcaf0) at curl.cpp:2920 #10 0x0000564e7c852e17 in S3fsCurl::CheckIAMCredentialUpdate () at curl.cpp:1905 #11 0x0000564e7c852e5d in S3fsCurl::insertAuthHeaders (this=this@entry=0x7f027c05e4a0) at curl.cpp:2812 #12 0x0000564e7c853b58 in S3fsCurl::RequestPerform (this=this@entry=0x7f027c05e4a0, dontAddAuthHeaders=dontAddAuthHeaders@entry=false) at curl.cpp:2392 #13 0x0000564e7c85c5f0 in S3fsMultiCurl::RequestPerformWrapper (arg=0x7f027c05e4a0) at curl.cpp:4479 #14 0x00007f028e9d9e65 in start_thread (arg=0x7f0282ffd700) at pthread_create.c:307 #15 0x00007f028e70288d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 ``` As usual, running it with valgrind didn't fail: ``` ==394== Memcheck, a memory error detector ==394== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==394== Using Valgrind-3.14.0-353a3587bb-20181007X and LibVEX; rerun with -h for copyright info ==394== Command: s3fs -f mybucket /mybucket-o rw,allow_other,iam_role=myrole,use_sse=1,default_acl=bucket-owner-full-control,url=https://s3.us-east-1.amazonaws.com,endpoint=us-east-1,use_path_request_style,use_cache=/tmp/s3fscache,del_cache,dev,suid ==394== Parent PID: 1 ==394== --394-- --394-- Valgrind options: --394-- -v --394-- --tool=memcheck --394-- --leak-check=full --394-- --show-leak-kinds=all --394-- --log-file=/s3fs.valgrind --394-- Contents of /proc/version: --394-- Linux version 4.14.152-127.182.amzn2.x86_64 (mockbuild@ip-10-0-1-129) (gcc version 7.3.1 20180712 (Red Hat 7.3.1-6) (GCC)) #1 SMP Thu Nov 14 17:32:43 UTC 2019 --394-- --394-- Arch and hwcaps: AMD64, LittleEndian, amd64-cx16-lzcnt-rdtscp-sse3-avx-avx2-bmi --394-- Page sizes: currently 4096, max supported 4096 --394-- Valgrind library directory: /usr/lib64/valgrind --394-- Reading syms from /usr/bin/s3fs --394-- Considering /usr/lib/debug/.build-id/45/69b66a26f0451df4e4958af1f15ee1dca16d9a.debug .. --394-- .. build-id is valid --394-- Warning: cross-CU LIMITATION: some inlined fn names --394-- might be shown as UnknownInlinedFun --394-- Reading syms from /usr/lib64/ld-2.17.so --394-- Considering /usr/lib/debug/.build-id/5c/c1a53b747a7e4d21198723c2b633e54f3c06d9.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/valgrind/memcheck-amd64-linux --394-- object doesn't have a symbol table --394-- object doesn't have a dynamic symbol table --394-- Scheduler: using generic scheduler lock implementation. --394-- Reading suppressions file: /usr/lib64/valgrind/default.supp ==394== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-394-by-???-on-sdu-rhn-synchronizer ==394== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-394-by-???-on-sdu-rhn-synchronizer ==394== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-394-by-???-on-sdu-rhn-synchronizer ==394== ==394== TO CONTROL THIS PROCESS USING vgdb (which you probably ==394== don't want to do, unless you know exactly what you're doing, ==394== or are doing some strange experiment): ==394== /usr/lib64/valgrind/../../bin/vgdb --pid=394 ...command... ==394== ==394== TO DEBUG THIS PROCESS USING GDB: start GDB like this ==394== /path/to/gdb s3fs ==394== and then give GDB the following command ==394== target remote | /usr/lib64/valgrind/../../bin/vgdb --pid=394 ==394== --pid is optional if only one valgrind process is running ==394== --394-- REDIR: 0x4019df0 (ld-linux-x86-64.so.2:strlen) redirected to 0x580c66c5 (???) --394-- REDIR: 0x4019bc0 (ld-linux-x86-64.so.2:index) redirected to 0x580c66df (???) --394-- Reading syms from /usr/lib64/valgrind/vgpreload_core-amd64-linux.so --394-- Reading syms from /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so ==394== WARNING: new redirection conflicts with existing -- ignoring it --394-- old: 0x04019df0 (strlen ) R-> (0000.0) 0x580c66c5 ??? --394-- new: 0x04019df0 (strlen ) R-> (2007.0) 0x04c2d0e0 strlen --394-- REDIR: 0x4019d70 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4c2e230 (strcmp) --394-- REDIR: 0x401aa30 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x4c31de0 (mempcpy) --394-- Reading syms from /usr/lib64/libfuse.so.2.9.2 --394-- Considering /usr/lib/debug/.build-id/01/3cd7d3f5015942bcd20e24a1ba945d1547af5b.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libcurl.so.4.3.0 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libxml2.so.2.9.1 --394-- Considering /usr/lib/debug/.build-id/3a/743aad8d50d50cd1dd10e272d620a1ca7e0f0f.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libcrypto.so.1.0.2k --394-- Considering /usr/lib/debug/.build-id/4c/f1939f660008cfa869d8364651f31aacd2c1c4.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libstdc++.so.6.0.19 --394-- Considering /usr/lib/debug/.build-id/e2/fb6e9c483d89e8e96d73c7ccf3e3a91e91bb81.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libm-2.17.so --394-- Considering /usr/lib/debug/.build-id/56/81c054fdabcf789f4dda66e94f1f6ed1747327.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libgcc_s-4.8.5-20150702.so.1 --394-- Considering /usr/lib/debug/.build-id/da/c0179f4555aefec9e97476201802fd20c03ec5.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libpthread-2.17.so --394-- Considering /usr/lib/debug/.build-id/8b/33f7f8c86f8d544c63c5541a8e42b3ddfef8b1.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libc-2.17.so --394-- Considering /usr/lib/debug/.build-id/39/8944d32cf16a67af51067a326e6c0cc14f90ed.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libdl-2.17.so --394-- Considering /usr/lib/debug/.build-id/18/113e6e83d8e981b8e8d808f7f3dbb23f950a1d.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libidn.so.11.6.11 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libssh2.so.1.0.1 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libssl3.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libsmime3.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libnss3.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libnssutil3.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libplds4.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libplc4.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libnspr4.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libgssapi_krb5.so.2.2 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libkrb5.so.3.3 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libk5crypto.so.3.1 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libcom_err.so.2.1 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/liblber-2.4.so.2.10.7 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libldap-2.4.so.2.10.7 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libz.so.1.2.7 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/liblzma.so.5.2.2 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libssl.so.1.0.2k --394-- Considering /usr/lib/debug/.build-id/3b/305c3ba17fe394862e749763f2956c9c890c2e.debug .. --394-- .. build-id is valid --394-- Considering /usr/lib/debug/.build-id/30/e5624c134dbd7e6f484dea8faf929a7e1317a3.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/librt-2.17.so --394-- Considering /usr/lib/debug/.build-id/47/49697bf078337576c4629f0d30b296a0939779.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libkrb5support.so.0.1 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libkeyutils.so.1.5 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libresolv-2.17.so --394-- Considering /usr/lib/debug/.build-id/30/09b26b33156eaaf99787aa3da0c6ae99649755.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libsasl2.so.3.0.0 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libselinux.so.1 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libcrypt-2.17.so --394-- Considering /usr/lib/debug/.build-id/84/467c988f41d853c58353beb247670e15da8bad.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libpcre.so.1.2.0 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libfreebl3.so --394-- object doesn't have a symbol table --394-- REDIR: 0x65770e0 (libc.so.6:strcasecmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6573e60 (libc.so.6:strnlen) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65793b0 (libc.so.6:strncasecmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65768c0 (libc.so.6:memset) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6576870 (libc.so.6:memcpy@GLIBC_2.2.5) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6573770 (libc.so.6:strcpy) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6573d30 (libc.so.6:strlen) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x657ba80 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6572020 (libc.so.6:strcat) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65722e0 (libc.so.6:strcmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65fc3b0 (libc.so.6:__memmove_chk) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x658ce90 (libc.so.6:strstr) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6572220 (libc.so.6:index) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6575c80 (libc.so.6:strspn) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65762a0 (libc.so.6:bcmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6573f40 (libc.so.6:strncmp) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x657d130 (libc.so.6:rawmemchr) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65757d0 (libc.so.6:strncpy) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6575810 (libc.so.6:rindex) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65758f0 (libc.so.6:strpbrk) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6576f40 (libc.so.6:stpcpy) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x65fc1a0 (libc.so.6:__memcpy_chk) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x658d9a0 (libc.so.6:strcasestr) redirected to 0x4a247a0 (_vgnU_ifunc_wrapper) --394-- REDIR: 0x6573d80 (libc.so.6:__GI_strlen) redirected to 0x4c2d040 (__GI_strlen) --394-- REDIR: 0x6575850 (libc.so.6:__GI_strrchr) redirected to 0x4c2caa0 (__GI_strrchr) --394-- REDIR: 0x656ca90 (libc.so.6:malloc) redirected to 0x4c29e1c (malloc) --394-- REDIR: 0x658c8d0 (libc.so.6:__GI_strstr) redirected to 0x4c32070 (__strstr_sse2) --394-- REDIR: 0x6575f50 (libc.so.6:memchr) redirected to 0x4c2e2d0 (memchr) --394-- REDIR: 0x657baf0 (libc.so.6:__GI_memcpy) redirected to 0x4c2eee0 (__GI_memcpy) --394-- REDIR: 0x6627c70 (libc.so.6:__strstr_sse42) redirected to 0x4c32100 (__strstr_sse42) --394-- REDIR: 0x656ceb0 (libc.so.6:free) redirected to 0x4c2af16 (free) --394-- REDIR: 0x6656630 (libc.so.6:__strlen_sse2_pminub) redirected to 0x4c2d020 (strlen) --394-- REDIR: 0x5b0aeb0 (libstdc++.so.6:operator new(unsigned long)) redirected to 0x4c2a43c (operator new(unsigned long)) --394-- REDIR: 0x663b9f0 (libc.so.6:__memcpy_ssse3_back) redirected to 0x4c2e6e0 (memcpy@@GLIBC_2.14) --394-- REDIR: 0x65762e0 (libc.so.6:__GI_memcmp) redirected to 0x4c30380 (__GI_memcmp) --394-- REDIR: 0x6625d40 (libc.so.6:__strcmp_sse42) redirected to 0x4c2e1e0 (__strcmp_sse42) --394-- REDIR: 0x656d4b0 (libc.so.6:calloc) redirected to 0x4c2bf23 (calloc) --394-- REDIR: 0xffffffffff600400 (???:???) redirected to 0x580c66b1 (???) --394-- REDIR: 0x6572260 (libc.so.6:__GI_strchr) redirected to 0x4c2cbd0 (__GI_strchr) --394-- REDIR: 0x656cf90 (libc.so.6:realloc) redirected to 0x4c2c0f5 (realloc) --394-- REDIR: 0x6573f80 (libc.so.6:__GI_strncmp) redirected to 0x4c2d870 (__GI_strncmp) --394-- REDIR: 0x6576f80 (libc.so.6:__GI_stpcpy) redirected to 0x4c30780 (__GI_stpcpy) --394-- REDIR: 0x6576a90 (libc.so.6:__GI_mempcpy) redirected to 0x4c31b10 (__GI_mempcpy) --394-- REDIR: 0x6581cc0 (libc.so.6:__strncpy_sse2_unaligned) redirected to 0x4c2d6b0 (__strncpy_sse2_unaligned) --394-- REDIR: 0x401ab80 (ld-linux-x86-64.so.2:stpcpy) redirected to 0x4c30b00 (stpcpy) --394-- REDIR: 0x5b0afc0 (libstdc++.so.6:operator new[](unsigned long)) redirected to 0x4c2aae1 (operator new[](unsigned long)) --394-- REDIR: 0x6626af0 (libc.so.6:__strncmp_sse42) redirected to 0x4c2d950 (__strncmp_sse42) --394-- REDIR: 0x6625c90 (libc.so.6:__strchr_sse42) redirected to 0x4c2cc90 (index) --394-- REDIR: 0x5b09020 (libstdc++.so.6:operator delete(void*)) redirected to 0x4c2b3c6 (operator delete(void*)) --394-- REDIR: 0x6650f60 (libc.so.6:__memcmp_sse4_1) redirected to 0x4c304c0 (__memcmp_sse4_1) --394-- REDIR: 0x65766e0 (libc.so.6:__GI_memmove) redirected to 0x4c31170 (__GI_memmove) --394-- REDIR: 0x6640f90 (libc.so.6:__memmove_ssse3_back) redirected to 0x4c2e390 (memcpy@GLIBC_2.2.5) --394-- REDIR: 0x6627ad0 (libc.so.6:__strrchr_sse42) redirected to 0x4c2cb30 (__strrchr_sse42) --394-- REDIR: 0x657d370 (libc.so.6:strchrnul) redirected to 0x4c31900 (strchrnul) --394-- REDIR: 0x6582ca0 (libc.so.6:__stpcpy_sse2_unaligned) redirected to 0x4c30940 (__stpcpy_sse2_unaligned) --394-- REDIR: 0x6581690 (libc.so.6:__strcpy_sse2_unaligned) redirected to 0x4c2d100 (strcpy) --394-- REDIR: 0x663b9e0 (libc.so.6:__memcpy_chk_ssse3_back) redirected to 0x4c31ed0 (__memcpy_chk) --394-- REDIR: 0x657d160 (libc.so.6:__GI___rawmemchr) redirected to 0x4c31960 (__GI___rawmemchr) --394-- REDIR: 0x65737b0 (libc.so.6:__GI_strcpy) redirected to 0x4c2d1e0 (__GI_strcpy) --394-- REDIR: 0x662f750 (libc.so.6:__strncasecmp_avx) redirected to 0x4c2daa0 (strncasecmp) --394-- REDIR: 0x6576920 (libc.so.6:__GI_memset) redirected to 0x4c30d30 (memset) --394-- REDIR: 0x662e0e0 (libc.so.6:__strcasecmp_avx) redirected to 0x4c2d9c0 (strcasecmp) --394-- REDIR: 0x5b09050 (libstdc++.so.6:operator delete[](void*)) redirected to 0x4c2ba38 (operator delete[](void*)) --394-- REDIR: 0x6572320 (libc.so.6:__GI_strcmp) redirected to 0x4c2e140 (__GI_strcmp) --394-- Reading syms from /usr/lib64/libnss_files-2.17.so --394-- Considering /usr/lib/debug/.build-id/a1/db0e8103de9f2540788eea6cbce3f639c2b39d.debug .. --394-- .. build-id is valid --394-- Reading syms from /usr/lib64/libnss_dns-2.17.so --394-- Considering /usr/lib/debug/.build-id/e9/451101125dc601a139a6b9c649f348e264aded.debug .. --394-- .. build-id is valid --394-- REDIR: 0xffffffffff600000 (???:???) redirected to 0x580c66a7 (???) --394-- Reading syms from /usr/lib64/libsoftokn3.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libsqlite3.so.0.8.6 --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libfreeblpriv3.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libnsssysinit.so --394-- object doesn't have a symbol table --394-- Reading syms from /usr/lib64/libnsspem.so --394-- object doesn't have a symbol table --394-- REDIR: 0x6580bf0 (libc.so.6:memrchr) redirected to 0x4c2e350 (memrchr) ```
Author
Owner

@ggtakec commented on GitHub (Apr 20, 2020):

This core dump may have thrown an exception from the s3fs_strtoofft function.
(I will PR & merge the modified version as soon as the correspondence of another PR is merged, so please wait a moment)

However, I think there is core reason why s3fs_strtoofft is throwing an exception, so it is necessary to investigate after fixing core dump.
Probably failed to get the Expiration(expiration) field in parsing the IAM credential response.
(The field name may be different or the wrong field name may have been set.)

<!-- gh-comment-id:616475754 --> @ggtakec commented on GitHub (Apr 20, 2020): This core dump may have thrown an exception from the s3fs_strtoofft function. (I will PR & merge the modified version as soon as the correspondence of another PR is merged, so please wait a moment) However, I think there is core reason why s3fs_strtoofft is throwing an exception, so it is necessary to investigate after fixing core dump. Probably failed to get the Expiration(expiration) field in parsing the IAM credential response. (The field name may be different or the wrong field name may have been set.)
Author
Owner

@gaul commented on GitHub (Jun 4, 2020):

@elstak can you test with the latest master?

<!-- gh-comment-id:638721530 --> @gaul commented on GitHub (Jun 4, 2020): @elstak can you test with the latest master?
Author
Owner

@gaul commented on GitHub (Aug 1, 2020):

Closing due to inactivity. Please reopen if symptoms persist.

<!-- gh-comment-id:667535587 --> @gaul commented on GitHub (Aug 1, 2020): Closing due to inactivity. Please reopen if symptoms persist.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#673
No description provided.