[GH-ISSUE #805] Segfault error 4 in libc-2.23.so #464

Closed
opened 2026-03-04 01:45:50 +03:00 by kerem · 6 comments
Owner

Originally created by @dystopian1 on GitHub (Jul 30, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/805

Additional Information

Hosted in AWS EC2:

We have been facing this issue for some time, although I do not have historical backtraces aside from the one posted in the details section which is very recent and after upgrading to the lastest fuse and s3fs versions. I can say that we did attempt to set multireq_max to a low number, and it appeared to have no effect.

We have another AWS EC2 instance with the following versions connecting to the same bucket using a slightly different command for the connection method, and this connection has not dropped in months:

Amazon Simple Storage Service File System V1.82(commit:566961c) with OpenSSL
2.9.4
4.4.0-1054-aws
Ubuntu 16.04.4 LTS \n \l
/usr/bin/s3fs bucket-name -o use_cache=/tmp -o allow_other -o multireq_max=75 -o default_acl=public-read-write -o parallel_count=10 /var/lib/myS3Bucket

The above is our initial instance first using s3fs. We then had to manually rebuild a secondary instance due to time constraints vs creating a Snapshot and restoring to a new instance. This resulted in different kernel versioning. Here are the versions with the instance having issues:

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.84(commit:06032aa) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse)

2.9.4

System information (uname -r)

4.4.0-1061-aws

Distro (cat /etc/issue)

Ubuntu 16.04.4 LTS \n \l

s3fs command line used (if applicable)

s3fs bucket-name -o allow_other -o multireq_max=75 -o default_acl=public-read-write -o parallel_count=10 /var/lib/myS3Bucket

/etc/fstab entry (if applicable):

s3fs syslog messages (grep s3fs /var/log/syslog, or s3fs outputs)

kernel: [3363383.180346] s3fs[19047]: segfault at 46f9da0 ip 00007f2b6edaa156 sp 00007f2accff0ae8 error 4 in libc-2.23.so[7f2b6ec5c000+1c0000]

### Details about issue

Backtrace:

Core was generated by `s3fs bucket-name -o allow_other -o multireq_max=75 -o default_acl=public-read-'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:238
238     ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory.
[Current thread is 1 (Thread 0x7f2accff1700 (LWP 19047))]
(gdb) bt
#0  __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:238
#1  0x00007f2b6e0ed1d5 in gnutls_buffer_append_data () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#2  0x00007f2b6e0e2b8b in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#3  0x00007f2b6e0dea73 in gnutls_session_set_data () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#4  0x00007f2b7002b305 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#5  0x00007f2b7002c0c0 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#6  0x00007f2b6ffe991d in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#7  0x00007f2b6fffaa81 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#8  0x00007f2b7000ef3e in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#9  0x00007f2b7000fb06 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#10 0x00007f2b7000681b in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#11 0x0000000000431051 in S3fsCurl::RequestPerform (this=0x7f2a8065c170) at curl.cpp:2051
#12 0x00000000004326a9 in S3fsMultiCurl::RequestPerformWrapper (arg=<optimized out>) at curl.cpp:4049
#13 0x00007f2b6f02d6ba in start_thread (arg=0x7f2accff1700) at pthread_create.c:333
#14 0x00007f2b6ed6341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109


 
Originally created by @dystopian1 on GitHub (Jul 30, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/805 ### Additional Information Hosted in AWS EC2: We have been facing this issue for some time, although I do not have historical backtraces aside from the one posted in the details section which is very recent and after upgrading to the lastest fuse and s3fs versions. I can say that we did attempt to set multireq_max to a low number, and it appeared to have no effect. We have another AWS EC2 instance with the following versions connecting to the same bucket using a slightly different command for the connection method, and this connection has not dropped in months: Amazon Simple Storage Service File System V1.82(commit:566961c) with OpenSSL 2.9.4 4.4.0-1054-aws Ubuntu 16.04.4 LTS \n \l /usr/bin/s3fs bucket-name -o use_cache=/tmp -o allow_other -o multireq_max=75 -o default_acl=public-read-write -o parallel_count=10 /var/lib/myS3Bucket The above is our initial instance first using s3fs. We then had to manually rebuild a secondary instance due to time constraints vs creating a Snapshot and restoring to a new instance. This resulted in different kernel versioning. Here are the versions with the instance having issues: #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.84(commit:06032aa) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse) 2.9.4 #### System information (uname -r) 4.4.0-1061-aws #### Distro (cat /etc/issue) Ubuntu 16.04.4 LTS \n \l #### s3fs command line used (if applicable) s3fs bucket-name -o allow_other -o multireq_max=75 -o default_acl=public-read-write -o parallel_count=10 /var/lib/myS3Bucket #### /etc/fstab entry (if applicable): ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, or s3fs outputs) kernel: [3363383.180346] s3fs[19047]: segfault at 46f9da0 ip 00007f2b6edaa156 sp 00007f2accff0ae8 error 4 in libc-2.23.so[7f2b6ec5c000+1c0000] ``` ### Details about issue Backtrace: Core was generated by `s3fs bucket-name -o allow_other -o multireq_max=75 -o default_acl=public-read-'. Program terminated with signal SIGSEGV, Segmentation fault. #0 __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:238 238 ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S: No such file or directory. [Current thread is 1 (Thread 0x7f2accff1700 (LWP 19047))] (gdb) bt #0 __memcpy_avx_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-avx-unaligned.S:238 #1 0x00007f2b6e0ed1d5 in gnutls_buffer_append_data () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #2 0x00007f2b6e0e2b8b in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #3 0x00007f2b6e0dea73 in gnutls_session_set_data () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #4 0x00007f2b7002b305 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #5 0x00007f2b7002c0c0 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #6 0x00007f2b6ffe991d in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #7 0x00007f2b6fffaa81 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #8 0x00007f2b7000ef3e in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #9 0x00007f2b7000fb06 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #10 0x00007f2b7000681b in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #11 0x0000000000431051 in S3fsCurl::RequestPerform (this=0x7f2a8065c170) at curl.cpp:2051 #12 0x00000000004326a9 in S3fsMultiCurl::RequestPerformWrapper (arg=<optimized out>) at curl.cpp:4049 #13 0x00007f2b6f02d6ba in start_thread (arg=0x7f2accff1700) at pthread_create.c:333 #14 0x00007f2b6ed6341d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
kerem closed this issue 2026-03-04 01:45:50 +03:00
Author
Owner

@dystopian1 commented on GitHub (Aug 2, 2018):

New Core Dump Trace:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/s3fs bucket-name -o allow_other -o multireq_max=75 -o default_acl=pub'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f31e467a428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f31697fa700 (LWP 26699))]
(gdb) bt
#0  0x00007f31e467a428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f31e467c02a in __GI_abort () at abort.c:89
#2  0x00007f31e46bc7ea in __libc_message (do_abort=do_abort@entry=2,
    fmt=fmt@entry=0x7f31e47d5ed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007f31e46c537a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f31e47d2cc7 "free(): invalid size",
    action=3) at malloc.c:5006
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3867
#5  0x00007f31e46c953c in __GI___libc_free (mem=<optimized out>) at malloc.c:2968
#6  0x00007f31e3b425c8 in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#7  0x00007f31e3b42f56 in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#8  0x00007f31e3ac3e65 in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#9  0x00007f31e3abfa78 in gnutls_handshake () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30
#10 0x00007f31e5a12e2d in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#11 0x00007f31e5a14332 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#12 0x00007f31e5a150c0 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#13 0x00007f31e59d240e in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#14 0x00007f31e59f7ef6 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#15 0x00007f31e59f8b06 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#16 0x00007f31e59ef81b in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4
#17 0x0000000000431051 in S3fsCurl::RequestPerform (this=0x7f31c0a58fd0) at curl.cpp:2051
#18 0x00000000004326a9 in S3fsMultiCurl::RequestPerformWrapper (arg=<optimized out>) at curl.cpp:4049
#19 0x00007f31e4a166ba in start_thread (arg=0x7f31697fa700) at pthread_create.c:333
#20 0x00007f31e474c41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
<!-- gh-comment-id:410004327 --> @dystopian1 commented on GitHub (Aug 2, 2018): New Core Dump Trace: ``` Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `/usr/bin/s3fs bucket-name -o allow_other -o multireq_max=75 -o default_acl=pub'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f31e467a428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory. [Current thread is 1 (Thread 0x7f31697fa700 (LWP 26699))] (gdb) bt #0 0x00007f31e467a428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54 #1 0x00007f31e467c02a in __GI_abort () at abort.c:89 #2 0x00007f31e46bc7ea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7f31e47d5ed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175 #3 0x00007f31e46c537a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7f31e47d2cc7 "free(): invalid size", action=3) at malloc.c:5006 #4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3867 #5 0x00007f31e46c953c in __GI___libc_free (mem=<optimized out>) at malloc.c:2968 #6 0x00007f31e3b425c8 in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #7 0x00007f31e3b42f56 in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #8 0x00007f31e3ac3e65 in ?? () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #9 0x00007f31e3abfa78 in gnutls_handshake () from /usr/lib/x86_64-linux-gnu/libgnutls.so.30 #10 0x00007f31e5a12e2d in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #11 0x00007f31e5a14332 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #12 0x00007f31e5a150c0 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #13 0x00007f31e59d240e in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #14 0x00007f31e59f7ef6 in ?? () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #15 0x00007f31e59f8b06 in curl_multi_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #16 0x00007f31e59ef81b in curl_easy_perform () from /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 #17 0x0000000000431051 in S3fsCurl::RequestPerform (this=0x7f31c0a58fd0) at curl.cpp:2051 #18 0x00000000004326a9 in S3fsMultiCurl::RequestPerformWrapper (arg=<optimized out>) at curl.cpp:4049 #19 0x00007f31e4a166ba in start_thread (arg=0x7f31697fa700) at pthread_create.c:333 #20 0x00007f31e474c41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ```
Author
Owner

@MrMoose commented on GitHub (Aug 6, 2018):

Same here after update to 1.84. Wasn't the case with 1.83

<!-- gh-comment-id:410649455 --> @MrMoose commented on GitHub (Aug 6, 2018): Same here after update to 1.84. Wasn't the case with 1.83
Author
Owner

@gaul commented on GitHub (Jan 24, 2019):

This looks like memory corruption possibly due to double free. Could you run s3fs under Valgrind which could reveal the cause of these symptoms?

<!-- gh-comment-id:457032293 --> @gaul commented on GitHub (Jan 24, 2019): This looks like memory corruption possibly due to double `free`. Could you run s3fs under Valgrind which could reveal the cause of these symptoms?
Author
Owner

@gaul commented on GitHub (Mar 15, 2019):

Could you test again with 1.85 which includes f53503438c?

<!-- gh-comment-id:473197152 --> @gaul commented on GitHub (Mar 15, 2019): Could you test again with 1.85 which includes f53503438c28910a36fe2b155d6623c9af41d951?
Author
Owner

@gaul commented on GitHub (Apr 9, 2019):

Closing due to inactivity. Please reopen if symptoms persist.

<!-- gh-comment-id:481182974 --> @gaul commented on GitHub (Apr 9, 2019): Closing due to inactivity. Please reopen if symptoms persist.
Author
Owner

@ebarault commented on GitHub (Jul 17, 2019):

for coming readers:
release 1.85 fixed this issue for me

<!-- gh-comment-id:512358321 --> @ebarault commented on GitHub (Jul 17, 2019): **for coming readers:** release 1.85 fixed this issue for me
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#464
No description provided.