[PR #1666] [MERGED] Make pseudo fd for each file opening and each pseudo fd has own upload info #2103

New issue

Closed

opened 2026-03-04 02:03:43 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:03:43 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/1666
Author: @ggtakec
Created: 5/23/2021
Status: ✅ Merged
Merged: 6/4/2021
Merged by: @gaul

Base: master ← Head: isolation_fd

📝 Commits (2)

57c9935 Introduced pseudo fd and separated fd for each file opening
f87f366 Added info object about multipart uploading for each pseudo fd

📊 Changes

15 files changed (+1251 additions, -572 deletions)

View changed files

📝 src/Makefile.am (+2 -0)
📝 src/curl.cpp (+4 -65)
📝 src/curl.h (+1 -2)
📝 src/fdcache.cpp (+97 -64)
📝 src/fdcache.h (+6 -5)
📝 src/fdcache_auto.cpp (+44 -23)
📝 src/fdcache_auto.h (+8 -4)
📝 src/fdcache_entity.cpp (+464 -314)
📝 src/fdcache_entity.h (+22 -22)
➕ src/fdcache_fdinfo.cpp (+243 -0)
➕ src/fdcache_fdinfo.h (+96 -0)
➕ src/fdcache_pseudofd.cpp (+134 -0)
➕ src/fdcache_pseudofd.h (+65 -0)
📝 src/s3fs.cpp (+60 -73)
📝 src/types.h (+5 -0)

📄 Description

Relevant Issue (if applicable)

n/a

Details

This PR is a fix for managing file descriptors that will be needed for future fixes in s3fs.
I'm improving the performance of s3fs and want to update the correction part in small pieces, so first I will post the code without upload logic change.

NOTE

This PR has a lot of modified code, but it is separated into the following two commits and put together as one PR.
The reason for posting them together is that if these are posted two separately, we may not understand the purpose of the PR.
Please do not squash when merging.

Current code

When a file(S3 object) is opened, the file descriptor(issued by the system(os)) to the local file of the cache file(or temporary file) managed by s3fs is returned to FUSE.
s3fs always returns the same file descriptor, even if the file is opened at the same time.
(However, the number of open is kept as the number of references)
And s3fs cannot distinguish system calls(and callers) using fd for the same file.

[Commit] Introduced pseudo fd and separated fd for each file opening

Changed s3fs to return a different pseudo file descriptor each time a file is opened at the same time.
This allows s3fs to recognize between callers, and system calls with fd can be processed depending on the caller.
Also, it has been fixed so that the file open flag(read/write, etc.) can be retained.

Note that there is only one cache file(and temporary file), even if the files are opened at the same time and different pseudo fd are issued.
In other words, pseuto fd is issued as a reference to one physical file.

[Commit] Added info object about multipart uploading for each pseudo fd

The multi-part upload id and etag managed by the FdEntity object have been changed to be managed in units of pseudo fd.
I changed the information of multipart upload to be managed individually, but please note that there is only one cache file(or temporary file) for one file.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/1666 **Author:** [@ggtakec](https://github.com/ggtakec) **Created:** 5/23/2021 **Status:** ✅ Merged **Merged:** 6/4/2021 **Merged by:** [@gaul](https://github.com/gaul) **Base:** `master` ← **Head:** `isolation_fd` --- ### 📝 Commits (2) - [`57c9935`](https://github.com/s3fs-fuse/s3fs-fuse/commit/57c9935829ee4189266a042812d6340d5c4a4838) Introduced pseudo fd and separated fd for each file opening - [`f87f366`](https://github.com/s3fs-fuse/s3fs-fuse/commit/f87f36679ee7087d2e066985569175fd67ae19ce) Added info object about multipart uploading for each pseudo fd ### 📊 Changes **15 files changed** (+1251 additions, -572 deletions) <details> <summary>View changed files</summary> 📝 `src/Makefile.am` (+2 -0) 📝 `src/curl.cpp` (+4 -65) 📝 `src/curl.h` (+1 -2) 📝 `src/fdcache.cpp` (+97 -64) 📝 `src/fdcache.h` (+6 -5) 📝 `src/fdcache_auto.cpp` (+44 -23) 📝 `src/fdcache_auto.h` (+8 -4) 📝 `src/fdcache_entity.cpp` (+464 -314) 📝 `src/fdcache_entity.h` (+22 -22) ➕ `src/fdcache_fdinfo.cpp` (+243 -0) ➕ `src/fdcache_fdinfo.h` (+96 -0) ➕ `src/fdcache_pseudofd.cpp` (+134 -0) ➕ `src/fdcache_pseudofd.h` (+65 -0) 📝 `src/s3fs.cpp` (+60 -73) 📝 `src/types.h` (+5 -0) </details> ### 📄 Description ## Relevant Issue (if applicable) n/a ## Details This PR is a fix for managing file descriptors that will be needed for future fixes in s3fs. _I'm improving the performance of s3fs and want to update the correction part in small pieces, so first I will post the code without upload logic change._ ### NOTE This PR has a lot of modified code, but it is separated into the following two commits and put together as one PR. The reason for posting them together is that if these are posted two separately, we may not understand the purpose of the PR. Please do not squash when merging. ### Current code When a file(S3 object) is opened, the file descriptor(issued by the system(os)) to the local file of the cache file(or temporary file) managed by s3fs is returned to FUSE. s3fs always returns the same file descriptor, even if the file is opened at the same time. (However, the number of open is kept as the number of references) And s3fs cannot distinguish system calls(and callers) using fd for the same file. ### [Commit] Introduced pseudo fd and separated fd for each file opening Changed s3fs to return a different pseudo file descriptor each time a file is opened at the same time. This allows s3fs to recognize between callers, and system calls with fd can be processed depending on the caller. Also, it has been fixed so that the file open flag(read/write, etc.) can be retained. Note that there is only one cache file(and temporary file), even if the files are opened at the same time and different pseudo fd are issued. In other words, pseuto fd is issued as a reference to one physical file. ### [Commit] Added info object about multipart uploading for each pseudo fd The multi-part upload id and etag managed by the FdEntity object have been changed to be managed in units of pseudo fd. I changed the information of multipart upload to be managed individually, but please note that there is only one cache file(or temporary file) for one file. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>