[GH-ISSUE #2090] Reading file is blocked if there is a dd command running on a s3fs mounted volume

kerem commented

2026-03-04 01:51:02 +03:00

Owner

Originally created by @creeew on GitHub (Jan 8, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2090

Additional Information

Version of s3fs being used (`s3fs --version`)

V1.91

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

2.9.2

Kernel information (`uname -r`)

4.19.113-300.el7.x86_64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

CentOS7

How to run s3fs, if applicable

s3fs test-bucket /mnt/dir -o rw,allow_other,no_check_certificate,use_path_request_style,disable_noobj_cache,url=http://x.x.x.x:9000,passwd_file=/.passwd-s3fs,dev,suid

Details about issue

Using s3fs command mount bucket "test-bucket" to dir "/mnt/dir". The bucket "test-bucket" has a object which name is "test.txt".
And change directory to /mnt/dir, running command dd if=/dev/zero of=big-file bs=1024 count=102400.
Try to read existed file "test.txt" while dd command is running.
The isssue is that "test.txt" reading operation was blocked, cannot get the file's content until dd command finished. Further more, the s3fs process memory was continuously growing.

Looks like s3fs reading operation "GetExistFdEntity" have to get the lock which was holded by "dd", maybe that's the reason why reading was blocked.
github.com/s3fs-fuse/s3fs-fuse@222110e153/src/fdcache.cpp (L626-L634)

Originally created by @creeew on GitHub (Jan 8, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2090  ### Additional Information #### Version of s3fs being used (`s3fs --version`) V1.91 #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 2.9.2 #### Kernel information (`uname -r`) 4.19.113-300.el7.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) CentOS7 #### How to run s3fs, if applicable s3fs test-bucket /mnt/dir -o rw,allow_other,no_check_certificate,use_path_request_style,disable_noobj_cache,url=http://x.x.x.x:9000,passwd_file=/.passwd-s3fs,dev,suid ### Details about issue Using s3fs command mount bucket "test-bucket" to dir "/mnt/dir". The bucket "test-bucket" has a object which name is "test.txt". And change directory to /mnt/dir, running command `dd if=/dev/zero of=big-file bs=1024 count=102400`. Try to read existed file "test.txt" while dd command is running. The isssue is that "test.txt" reading operation was blocked, cannot get the file's content until dd command finished. Further more, the s3fs process memory was continuously growing. Looks like s3fs reading operation "GetExistFdEntity" have to get the lock which was holded by "dd", maybe that's the reason why reading was blocked. https://github.com/s3fs-fuse/s3fs-fuse/blob/222110e153248e85030ceb59501c2c73edc0f255/src/fdcache.cpp#L626-L634

kerem commented

2026-03-04 01:51:03 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 11, 2023):

@creeew Thanks for this Issue post.
I just wanted to inform you about I'm going to investigate this matter soon.

@ggtakec commented on GitHub (Jan 11, 2023): @creeew Thanks for this Issue post. I just wanted to inform you about I'm going to investigate this matter soon.

kerem commented

2026-03-04 01:51:03 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 12, 2023):

@creeew
I've checked both the master branch code and the v1.91 code, just like you, but I haven't been able to reproduce the same problem.
In each case, I tried specifying and not specifying the use_path_request_style option, but I could not reproduce it.

Locking in the FdManager::GetExistFdEntity() function is just a linear search of the fdent_map_t variable (I believe it is finite).
So it's hard to imagine blocking during the upload of a large file with the dd command.
And I'm also concerned that the used memory keeps increasing as you point.

Therefore, I have not yet determined whether this problem depends on the individual environment or is due to a flaw in the code.

If you can, could you try same test with the code in the master branch?
We have fixed some bugs since the v1.91 release, so it may show different results.
Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 12, 2023): @creeew I've checked both the master branch code and the v1.91 code, just like you, but I haven't been able to reproduce the same problem. _In each case, I tried specifying and not specifying the `use_path_request_style` option, but I could not reproduce it._ Locking in the `FdManager::GetExistFdEntity()` function is just a linear search of the `fdent_map_t variable` (I believe it is finite). So it's hard to imagine blocking during the upload of a large file with the `dd` command. And I'm also concerned that the used memory keeps increasing as you point. Therefore, I have not yet determined whether this problem depends on the individual environment or is due to a flaw in the code. If you can, could you try same test with the code in the master branch? We have fixed some bugs since the v1.91 release, so it may show different results. Thanks in advance for your help.

kerem commented

2026-03-04 01:51:03 +03:00

Author

Owner

@creeew commented on GitHub (Jan 13, 2023):

@ggtakec Thank you for your response.
I could reproduce the same issue in my environment with master branch code.
Here are steps to reproduce:

Mount bucket to dir /mnt/test
s3fs test-bucket: /mnt/test -o rw,allow_other,no_check_certificate,use_path_request_style,disable_noobj_cache,url=http://127.0.0.1:9000,passwd_file=/etc/.passwd-s3fs,dev,suid -f -d
Using dd command to create big file
dd if=/dev/zero of=/mnt/test/big bs=1M count=10000
Reading existed object in mounted dir
cat /mnt/test/existed_file.txt

It's hard to say we could be blocked in every single time, maybe you would be able to reproduce it if you try a few more times.

additional:
So sorry for that the maybe blocking codes I was mentioned before was incorrect.
github.com/s3fs-fuse/s3fs-fuse@16bc44948e/src/fdcache_entity.cpp (L365-L376)
I try to print log of AutoLock auto_lock(&fdent_lock, locktype); before and after, reading file will print before log directly and print after log in 4 or 5 minutes later.

@creeew commented on GitHub (Jan 13, 2023): @ggtakec Thank you for your response. I could reproduce the same issue in my environment with master branch code. Here are steps to reproduce: 1. Mount bucket to dir _/mnt/test_ `s3fs test-bucket: /mnt/test -o rw,allow_other,no_check_certificate,use_path_request_style,disable_noobj_cache,url=http://127.0.0.1:9000,passwd_file=/etc/.passwd-s3fs,dev,suid -f -d` 2. Using dd command to create big file `dd if=/dev/zero of=/mnt/test/big bs=1M count=10000 ` 3. Reading existed object in mounted dir `cat /mnt/test/existed_file.txt ` It's hard to say we could be blocked in every single time, maybe you would be able to reproduce it if you try a few more times. additional: So sorry for that the maybe blocking codes I was mentioned before was incorrect. https://github.com/s3fs-fuse/s3fs-fuse/blob/16bc44948ee2a4167dd2f585ba8ac7e3d5f7d63d/src/fdcache_entity.cpp#L365-L376 I try to print log of `AutoLock auto_lock(&fdent_lock, locktype);` before and after, reading file will print before log directly and print after log in 4 or 5 minutes later.

kerem commented

2026-03-04 01:51:03 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 15, 2023):

@creeew Thank you for the information, and I was able to reproduce almost the same issue.

The reading process will be blocked for a portion of the large file upload process, not for the entire time.

Uploading a (large) file with s3fs works like this:
Although there are some differences depending on the option.

First, when you create a file with copy, dd, etc., s3fs receives data from the system in block size units via FUSE.
s3fs accumulates the received data up to a certain number of bytes.
When the accumulated data reaches the size of the multipart upload or reaches the total number of bytes in the file, it starts transferring(uploading) to the S3 server.

This issue's problem is the blocking the read file in only last processing upload.
You will not be blocked while accumulating data.

[Reference] The following two sources are this problem codes:
https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/fdcache_entity.cpp#L1402-L1410
https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/fdcache_entity.cpp#L1402-L1410

Upload times may vary depending on s3fs options.
For example, if you specify the nomultipart option, it will accumulate the entire file size and upload the entire large file size, which will take a long time.
Conversely, if you allow multipart uploads, you will only be blocked while uploading the multipart upload size.
Therefore, I think that you can adjust the blocked period by adjusting the multipart size.

Currently, I think it is difficult to modify to remove this blocking area when uploading.
This exclusive control is performed to protect file information(used internally) for uploading, file descriptors, etc., and I feel that reducing the lock range is quite difficult.

I think you can instead adjust the s3fs options regarding multipart uploads(multipart_size, multipart_copy_size, multipart_threshold, singlepart_copy_limit, streamupload, etc) to make it block for less time.
Could you tune options and parameters to avoid the worst for you?

@ggtakec commented on GitHub (Jan 15, 2023): @creeew Thank you for the information, and I was able to reproduce almost the same issue. The reading process will be blocked for a portion of the large file upload process, not for the entire time. Uploading a (large) file with s3fs works like this: _Although there are some differences depending on the option._ First, when you create a file with copy, dd, etc., s3fs receives data from the system in block size units via FUSE. s3fs accumulates the received data up to a certain number of bytes. When the accumulated data reaches the size of the multipart upload or reaches the total number of bytes in the file, it starts transferring(uploading) to the S3 server. This issue's problem is the blocking the read file in only last processing upload. You will not be blocked while accumulating data. [Reference] The following two sources are this problem codes: https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/fdcache_entity.cpp#L1402-L1410 https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/fdcache_entity.cpp#L1402-L1410 Upload times may vary depending on s3fs options. For example, if you specify the `nomultipart` option, it will accumulate the entire file size and upload the entire large file size, which will take a long time. Conversely, if you allow multipart uploads, you will only be blocked while uploading the multipart upload size. Therefore, I think that you can adjust the blocked period by adjusting the multipart size. Currently, I think it is difficult to modify to remove this blocking area when uploading. This exclusive control is performed to protect file information(used internally) for uploading, file descriptors, etc., and I feel that reducing the lock range is quite difficult. I think you can instead adjust the s3fs options regarding multipart uploads(`multipart_size`, `multipart_copy_size`, `multipart_threshold`, `singlepart_copy_limit`, `streamupload`, etc) to make it block for less time. Could you tune options and parameters to avoid the worst for you?

kerem commented

2026-03-04 01:51:04 +03:00

Author

Owner

@huntersman commented on GitHub (Feb 15, 2023):

@ggtakec I have the same issue when upload large file.

And I noticed that when we upload large file (like 50 GB) continuously via S3FS, subsequent files will be blocked until previous files have finished multipart uploading.

I guess it is because MultiPerform wait for semaphore.
https://github.com/s3fs-fuse/s3fs-fuse/blob/e715b77307fe3a36cfa9a487ff2a5fa0b5cf421c/src/curl_multi.cpp#L180-L182

Is it possible that we change s3fs_flush to asynchronous?

Sorry I missed #367.

@huntersman commented on GitHub (Feb 15, 2023): @ggtakec I have the same issue when upload large file. And I noticed that when we upload large file (like 50 GB) continuously via S3FS, subsequent files will be blocked until previous files have finished multipart uploading. I guess it is because `MultiPerform` wait for semaphore. [https://github.com/s3fs-fuse/s3fs-fuse/blob/e715b77307fe3a36cfa9a487ff2a5fa0b5cf421c/src/curl_multi.cpp#L180-L182](https://github.com/s3fs-fuse/s3fs-fuse/blob/e715b77307fe3a36cfa9a487ff2a5fa0b5cf421c/src/curl_multi.cpp#L180-L182) Is it possible that we change `s3fs_flush` to asynchronous? --- Sorry I missed #367.

kerem commented

2026-03-04 01:51:04 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 19, 2023):

@huntersman
Waiting for the semaphore in S3fsMultiCurl::MultiPerform waits for the flush process to complete.
Currently, changing s3fs_flush to asynchronous is difficult, but you can try the streamupload option.
You can try this option on your master branch.
This option may reduce latency during flush.

@ggtakec commented on GitHub (Feb 19, 2023): @huntersman Waiting for the semaphore in `S3fsMultiCurl::MultiPerform` waits for the flush process to complete. Currently, changing `s3fs_flush` to asynchronous is difficult, but you can try the `streamupload` option. You can try this option on your master branch. This option may reduce latency during flush.

kerem commented

2026-03-04 01:51:04 +03:00

Author

Owner

@fangqianan commented on GitHub (Dec 11, 2024):

Hello, I met the same problem as @creeew . And I was wondering whether it's possible to search FdEntity in GetExistFdEntity via

fdent_map_t::iterator iter = fent.find(std::string(path));
if(fent.end() == iter && !force_tmpfile && !FdManager::IsCacheDir()){
    for(iter = fent.begin(); iter != fent.end(); ++iter){
        if(iter->second && iter->second->IsOpen() && 0 == strcmp(iter->second->GetPath(), path)){
            break;      // found opened fd in mapping
        }
    }
}

instead of traversing the fent and acquiring fent_lock for every fent. Like OpenExistFdEntity does.

This way, there is no need to try to get every opened file's fent_lock, thus it will not block other reading req.

@fangqianan commented on GitHub (Dec 11, 2024): Hello, I met the same problem as @creeew . And I was wondering whether it's possible to search FdEntity in `GetExistFdEntity` via ```c++ fdent_map_t::iterator iter = fent.find(std::string(path)); if(fent.end() == iter && !force_tmpfile && !FdManager::IsCacheDir()){ for(iter = fent.begin(); iter != fent.end(); ++iter){ if(iter->second && iter->second->IsOpen() && 0 == strcmp(iter->second->GetPath(), path)){ break; // found opened fd in mapping } } } ``` instead of traversing the fent and acquiring fent_lock for every fent. Like `OpenExistFdEntity` does. This way, there is no need to try to get every opened file's fent_lock, thus it will not block other reading req.