mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 21:35:58 +03:00
[GH-ISSUE #2220] ThreadSanitizer data race at OPENSSL_sk_dup/free in test_concurrent_directory_updates test #1126
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#1126
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ggtakec on GitHub (Jul 16, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2220
Related Issues / PRs
#1997 #2211 #2209
Version of s3fs being used (
s3fs --version)v1.92
Detail
We've fixed the errors detected by the ThreadSanitizer test in #2211, #2209, but it still doesn't seem to be resolved.
But I'm still getting errors like this:
Similar to the above error,
Previous readhas also detected:From this error message there is a conflict in a function of libcrypto.
OpenSSL's multithreading support for this we already implemented in the
s3fs_init_crypt_mutexfunction.A hint about this error is that it appears to be occurring on data access that should be protected by
Mutex M0.(It is
S3fsCurl::callback_locks.ssl_sessionthat initialized in theS3fsCurl::InitS3fsCurlmethod)So the above error seems to be related to
CURL_LOCK_DATA_SSL_SESSIONwhich is one ofCURLSHOPT_SHARE.However, Already the lock and unlock functions are also registered, so I still don't understand why this error occurs.
Reproduce
This error occurs infrequently in the ThreadSanitizer test in test_concurrent_directory_updates.
Note
@gaul
I will merge #1997 into this issue.
Please allow me to discuss, investigate and report on this matter here.
@ggtakec commented on GitHub (May 27, 2024):
This my bug research is still in progress, but I would like to share the results.
We added support for
fedora:40in PR #2451, but theSanitize Threadcheck now always(almost) fails.The failure is mainly by the following
Data Race:The above is a call from
S3fsCurl::HeadRequest, which reports thatOPENSSL_sk_duphas accessed the freed area.(
S3fsCurl::HeadRequestis one example, and we have also confirmed cases such asS3fsCurl::GetObjectRequest.)There seemed to be two possibilities for this:
(1)
s3fs_sha256called fromS3fsCurl::insertV4Headers(2)
CURLSHOPT_SHAREWhen the error occurs, it occurs after calling
curl_easy_performin one of the threads.Therefore, (1) is not considered to be the cause.
s3fs specifies
CURL_LOCK_DATA_SSL_SESSIONin curl optionCURLSHOPT_SHAREas default.This error seems likely to be accessing data from an SSL session that has been updated or destroyed.
I tried starting s3fs with the
nosscache(andnodnscache) options and the error no longer occurs.(NOTE: However, the number of tests may not be sufficient yet)
I believe that the LOCK function related to
CURLSHOPT_SHAREis set correctly, but it seems that exclusive control of the internally shared data of the SSL session is not possible.The Sanitize Thread check (fedra:40) uses
OpenSSL 3.2.1 30 Jan 2024 (Library: OpenSSL 3.2.1 30 Jan 2024).@ggtakec commented on GitHub (Nov 30, 2024):
This issue was resolved by PR #2608 (final refactoring of #2600).