[PR #2348] [MERGED] Fixed a bug in the re-upload part of Streamupload #2493

New issue

Closed

opened 2026-03-04 02:05:50 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:05:50 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/2348
Author: @ggtakec
Created: 10/15/2023
Status: ✅ Merged
Merged: 11/10/2023
Merged by: @gaul

Base: master ← Head: fix_streamupload

📝 Commits (1)

7222af8 Fixed a bug in the re-upload part of Streamupload

📊 Changes

3 files changed (+34 additions, -6 deletions)

View changed files

📝 src/fdcache_entity.cpp (+23 -2)
📝 src/fdcache_fdinfo.cpp (+9 -2)
📝 src/fdcache_fdinfo.h (+2 -2)

📄 Description

Relevant Issue (if applicable)

n/a

Details

Problem phenomenon

The CI test was failing in rare cases in the test_multipart_mix test of StreamUpload.
(It seemed to happen more often on macos than on linux)

StreamUpload logic

In StreamUpload, if the part range of each multipart is filled with updates, that part will be uploaded first without waiting for flush.
This StreamUpload logic allows for re-uploads of the same part if there are updates(written) to the same part range after the first upload for the same part.
In this case, when flushed the file, this part will be re-uploaded and the previous upload will be overwritten(cancelled).
This is the usual expected StreamUpload's logic.

Behavior in case of error

The error detected this time occurred in a case where Flush was called and a re-upload was started when the previously uploaded part had not yet finished uploading.
It means that uploads of the same part range were running at the same time.
Normally, the upload that started first will complete first, and the ETag value will be overwritten by the update caused by the re-upload.
However, on this error case, the re-upload process completed first, and the earlier upload process completed later, resulting in an invalid (cancelled) ETag value.

Fixed bug

In order to avoid this, we have modified it so that if it is detected that the part to be re-uploaded is still being uploaded, the re-upload will start after the upload including that part is completed.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/2348 **Author:** [@ggtakec](https://github.com/ggtakec) **Created:** 10/15/2023 **Status:** ✅ Merged **Merged:** 11/10/2023 **Merged by:** [@gaul](https://github.com/gaul) **Base:** `master` ← **Head:** `fix_streamupload` --- ### 📝 Commits (1) - [`7222af8`](https://github.com/s3fs-fuse/s3fs-fuse/commit/7222af8ce4bf1b5060bdef58d7400b4b682d3ff5) Fixed a bug in the re-upload part of Streamupload ### 📊 Changes **3 files changed** (+34 additions, -6 deletions) <details> <summary>View changed files</summary> 📝 `src/fdcache_entity.cpp` (+23 -2) 📝 `src/fdcache_fdinfo.cpp` (+9 -2) 📝 `src/fdcache_fdinfo.h` (+2 -2) </details> ### 📄 Description ### Relevant Issue (if applicable) n/a ### Details #### Problem phenomenon The CI test was failing in rare cases in the `test_multipart_mix` test of StreamUpload. (It seemed to happen more often on `macos` than on `linux`) #### StreamUpload logic In StreamUpload, if the part range of each multipart is filled with updates, that part will be uploaded first without waiting for `flush`. This StreamUpload logic allows for re-uploads of the same part if there are updates(written) to the same part range after the first upload for the same part. In this case, when flushed the file, this part will be re-uploaded and the previous upload will be overwritten(cancelled). This is the usual expected StreamUpload's logic. #### Behavior in case of error The error detected this time occurred in a case where Flush was called and a re-upload was started when the previously uploaded part had not yet finished uploading. It means that uploads of the same part range were running at the same time. Normally, the upload that started first will complete first, and the ETag value will be overwritten by the update caused by the re-upload. However, on this error case, the re-upload process completed first, and the earlier upload process completed later, resulting in an invalid (cancelled) ETag value. #### Fixed bug In order to avoid this, we have modified it so that if it is detected that the part to be re-uploaded is still being uploaded, the re-upload will start after the upload including that part is completed. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>