[PR #1783] [MERGED] Add a test that is multi-block writing by one flush #2169

New issue

Closed

opened 2026-03-04 02:04:07 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:04:07 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/1783
Author: @ggtakec
Created: 10/24/2021
Status: ✅ Merged
Merged: 10/25/2021
Merged by: @gaul

Base: master ← Head: add_multiblock_write_test

📝 Commits (2)

1682a1d Add a test that is multi-block writing by one flush
36e3159 Fixed parse_string function in write_multiblock.cc

📊 Changes

3 files changed (+407 additions, -0 deletions)

View changed files

📝 test/Makefile.am (+3 -0)
📝 test/integration-test-main.sh (+138 -0)
➕ test/write_multiblock.cc (+266 -0)

📄 Description

Relevant Issue (if applicable)

n/a

Details

Add new one test that writes multiple blocks.

This test checks for caches, multipart upload boundaries, copy uploads in the presence of blank areas, and more.
The current test may be sufficient, but I would like to use it to check if it works properly even if the logic changes in the future.

Each write block is separated by an area that does not write to each other.
And after doing these writes, it is flushed to the file only once.
(Flush does not occur every time a block is written, it is flushed after writing all blocks)

The test code looks like this(simple):

Open the file(new or exist)
Write multiple blocks
Close the file(flush is called only once)

And perform the above tests on newly created files and existing files.
For existing files, do not change the existing file size. (no truncate, write only inside the existing file size)

Multiple blocks are the following three blocks:

Smaller than the minimum size for multipart uploads in the middle of a file
Larger than the minimum size for multipart uploads in the middle of a file
Size smaller than the minimum size for multipart upload to the end of the file(new creation only)

After writing is complete, if s3fs uses the cache directory, delete the cache files and compare files(cmp).

This is not a problem in the current code, but each write block is also a cross-border test with a minimum multipart size as the boundary.

About test sub program(`write_multiblock.cc`):

I tried to execute it with the dd command as follows, but since flush occurs after writing each block, I decided to use the test with C source.

exec 3<> test_file
dd if=input_file bs=$((1024 * 1024)) count=1           seek=12 >&3
dd if=input_file bs=$((1024 * 1024)) count=$((2 * 10)) seek=22 >&3
dd if=input_file bs=$((1024 * 1024)) count=1           seek=23 >&3
exec 3>&-

If you can do the same on a script basis, I consider changing it.
So far, I could only do it by C base code.
(In the case of python, there is a problem that it is the file is always truncated)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/1783 **Author:** [@ggtakec](https://github.com/ggtakec) **Created:** 10/24/2021 **Status:** ✅ Merged **Merged:** 10/25/2021 **Merged by:** [@gaul](https://github.com/gaul) **Base:** `master` ← **Head:** `add_multiblock_write_test` --- ### 📝 Commits (2) - [`1682a1d`](https://github.com/s3fs-fuse/s3fs-fuse/commit/1682a1d66117559d9105c6eca244c2f7da35130f) Add a test that is multi-block writing by one flush - [`36e3159`](https://github.com/s3fs-fuse/s3fs-fuse/commit/36e315998beea5dd26b1d722a77b6fba941ce449) Fixed parse_string function in write_multiblock.cc ### 📊 Changes **3 files changed** (+407 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `test/Makefile.am` (+3 -0) 📝 `test/integration-test-main.sh` (+138 -0) ➕ `test/write_multiblock.cc` (+266 -0) </details> ### 📄 Description ### Relevant Issue (if applicable) n/a ### Details Add new one test that writes multiple blocks. This test checks for caches, multipart upload boundaries, copy uploads in the presence of blank areas, and more. The current test may be sufficient, but I would like to use it to check if it works properly even if the logic changes in the future. Each write block is separated by an area that does not write to each other. And after doing these writes, it is flushed to the file only once. (Flush does not occur every time a block is written, it is flushed after writing all blocks) The test code looks like this(simple): - Open the file(new or exist) - Write multiple blocks - Close the file(`flush` is called only once) And perform the above tests on newly created files and existing files. For existing files, do not change the existing file size. (no truncate, write only inside the existing file size) Multiple blocks are the following three blocks: - Smaller than the minimum size for multipart uploads in the middle of a file - Larger than the minimum size for multipart uploads in the middle of a file - Size smaller than the minimum size for multipart upload to the end of the file(new creation only) After writing is complete, if s3fs uses the cache directory, delete the cache files and compare files(`cmp`). This is not a problem in the current code, but each write block is also a cross-border test with a minimum multipart size as the boundary. ##### About test sub program(`write_multiblock.cc`): I tried to execute it with the dd command as follows, but since flush occurs after writing each block, I decided to use the test with C source. ``` exec 3<> test_file dd if=input_file bs=$((1024 * 1024)) count=1 seek=12 >&3 dd if=input_file bs=$((1024 * 1024)) count=$((2 * 10)) seek=22 >&3 dd if=input_file bs=$((1024 * 1024)) count=1 seek=23 >&3 exec 3>&- ``` If you can do the same on a script basis, I consider changing it. So far, I could only do it by C base code. (In the case of python, there is a problem that it is the file is always truncated) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>