[GH-ISSUE #1355] Why do we use fdent_data_lock #729

New issue

Open

opened 2026-03-04 01:48:16 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:48:16 +03:00

Owner

Originally created by @lwpdsg on GitHub (Aug 10, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1355

Hi, When I testing s3fs_fuse I found the fuse has control the parallel operation of the same file .That means I can't read the file while somebody is writing it, seems like all parallel operation I made to same file will transfer to sequential function call when passing fuse. For instance, I add 10 secs's sleep in s3fs read then try to do some parallel read using multiple shell windows, when the process sleeps at the code I added, I used pstack to output the functions stacks . And there is only one s3fs_read, the next will be called after the current one was finished. so I am confused with these fdent_data_locks , what's the function of them ?

Originally created by @lwpdsg on GitHub (Aug 10, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1355 Hi, When I testing s3fs_fuse I found the fuse has control the parallel operation of the same file .That means I can't read the file while somebody is writing it, seems like all parallel operation I made to same file will transfer to sequential function call when passing fuse. For instance, I add 10 secs's sleep in s3fs read then try to do some parallel read using multiple shell windows, when the process sleeps at the code I added, I used pstack to output the functions stacks . And there is only one s3fs_read, the next will be called after the current one was finished. so I am confused with these fdent_data_locks , what's the function of them ?

kerem added the

performance

label

2026-03-04 01:48:16 +03:00

kerem commented

2026-03-04 01:48:17 +03:00

Author

Owner

@gaul commented on GitHub (Aug 11, 2020):

fdent_data_lock protects several members including pagelist using a simple per-fd lock. Previously there was a single lock for metadata and data for each fd which blocked also stat and readdir during uploads, #928. There are more complicated locking strategies, e.g., byte ranges, although we have not implemented those yet due to their complexity. One easier approach would be using a multi-reader, single-writer lock although this needs to make sure that s3fs does not download the same range concurrently, among other things.

@gaul commented on GitHub (Aug 11, 2020): `fdent_data_lock` protects several members including `pagelist` using a simple per-fd lock. Previously there was a single lock for metadata and data for each fd which blocked also `stat` and `readdir` during uploads, #928. There are more complicated locking strategies, e.g., byte ranges, although we have not implemented those yet due to their complexity. One easier approach would be using a multi-reader, single-writer lock although this needs to make sure that s3fs does not download the same range concurrently, among other things.

kerem commented

2026-03-04 01:48:17 +03:00

Author

Owner

@lwpdsg commented on GitHub (Aug 13, 2020):

fdent_data_lock protects several members including pagelist using a simple per-fd lock. Previously there was a single lock for metadata and data for each fd which blocked also stat and readdir during uploads, #928. There are more complicated locking strategies, e.g., byte ranges, although we have not implemented those yet due to their complexity. One easier approach would be using a multi-reader, single-writer lock although this needs to make sure that s3fs does not download the same range concurrently, among other things.

Thank you for your reply. I know there are two mutexes in the fdentity , the fdent_lock controls the metadata like fd, orgmeta. the other one which is fdent_data_lock controls the pagelist and data to limit concurrent reading/writing of the same file. But the question is fuse doesn't allow reading or writing the same file concurrently. Among our test , we try to read one file in multithreads. As expetced, one thread can get the fdent_data_lock in FdEntity::Read and return the data , all the others should block at somewhere like FdEntity::Open or FdEntity::Read while trying to get the fdent_data_lock. But the output of the pstack shows only one thread processed in s3fs_read，the fuse blocks the other thread of reading. So the fdent_data_lock seems useless since there won't be two threads operating the data of the file concurrently . Please let me know if anying I misunderstand, THX

@lwpdsg commented on GitHub (Aug 13, 2020): > `fdent_data_lock` protects several members including `pagelist` using a simple per-fd lock. Previously there was a single lock for metadata and data for each fd which blocked also `stat` and `readdir` during uploads, #928. There are more complicated locking strategies, e.g., byte ranges, although we have not implemented those yet due to their complexity. One easier approach would be using a multi-reader, single-writer lock although this needs to make sure that s3fs does not download the same range concurrently, among other things. Thank you for your reply. I know there are two mutexes in the fdentity , the fdent_lock controls the metadata like fd, orgmeta. the other one which is fdent_data_lock controls the pagelist and data to limit concurrent reading/writing of the same file. But the question is fuse doesn't allow reading or writing the same file concurrently. Among our test , we try to read one file in multithreads. As expetced, one thread can get the fdent_data_lock in FdEntity::Read and return the data , all the others should block at somewhere like FdEntity::Open or FdEntity::Read while trying to get the fdent_data_lock. But the output of the pstack shows only one thread processed in s3fs_read，the fuse blocks the other thread of reading. So the fdent_data_lock seems useless since there won't be two threads operating the data of the file concurrently . Please let me know if anying I misunderstand, THX

kerem referenced this issue

2026-03-04 02:01:21 +03:00

[PR #729] [MERGED] FreeBSD build fixes #1620