[GH-ISSUE #607] Improve random writes and appends #344

New issue

Closed

opened 2026-03-04 01:44:35 +03:00 by kerem · 4 comments

kerem commented

2026-03-04 01:44:35 +03:00

Owner

Originally created by @CAFxX on GitHub (May 24, 2017).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/607

In the README.md the performance of random writes and appends is listed as a limitation:

Limitations

Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:

random writes or appends to files require rewriting the entire file

If this is still the case, was it ever considered to use multipart upload, and specifically both Upload Part and Upload Part - Copy, to speed it up?

Specifically, an append operation would become (simplifying):

initiate multipart upload
upload part - copy (old object)
upload part (data to append)
complete multipart upload
delete old object

A random write operation (e.g. "helloworld" -> "hello World") would become:

initiate multipart upload
upload part - copy (old object, off 0 len 5 "hello")
upload part (changed data, " W")
upload part - copy (old object, off 6 len 4 "orld")
complete multipart upload
delete old object

This approach would probably be beneficial when the existing object is big and the amount of data to append or changed is small, so some heuristics may be needed. In addition, these heuristics should also consider that (as noted below by @sqlbot) the upload part API has some restrictions in terms of minimum/maximum allowable part size ("Part size: 5 MB to 5 GB, last part can be < 5 MB")

Originally created by @CAFxX on GitHub (May 24, 2017). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/607 In the README.md the performance of random writes and appends is listed as a limitation: > # Limitations > Generally S3 cannot offer the same performance or semantics as a local file system. More specifically: > - random writes or appends to files require rewriting the entire file If this is still the case, was it ever considered to use multipart upload, and specifically both [Upload Part](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPart.html) and [Upload Part - Copy](https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html), to speed it up? Specifically, an append operation would become (simplifying): - initiate multipart upload - upload part - copy (old object) - upload part (data to append) - complete multipart upload - delete old object A random write operation (e.g. "helloworld" -> "hello World") would become: - initiate multipart upload - upload part - copy (old object, off 0 len 5 "hello") - upload part (changed data, " W") - upload part - copy (old object, off 6 len 4 "orld") - complete multipart upload - delete old object This approach would probably be beneficial when the existing object is big and the amount of data to append or changed is small, so some heuristics may be needed. In addition, these heuristics should also consider that (as noted below by @sqlbot) the upload part API has some restrictions in terms of minimum/maximum allowable part size ("Part size: 5 MB to 5 GB, last part can be < 5 MB")

kerem closed this issue

2026-03-04 01:44:35 +03:00

kerem commented

2026-03-04 01:44:36 +03:00

Author

Owner

@sqlbot commented on GitHub (May 25, 2017):

S3 itself would not allow such an operation at small scale. The Multipart Upload API, whether the part data is uploaded or copied from another object, requires that each part (except the last part) have a minimum size of 5 MB.

http://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

@sqlbot commented on GitHub (May 25, 2017): S3 itself would not allow such an operation at small scale. The Multipart Upload API, whether the part data is uploaded or copied from another object, requires that each part (except the last part) have a minimum size of 5 MB. http://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

kerem commented

2026-03-04 01:44:36 +03:00

Author

Owner

@CAFxX commented on GitHub (May 25, 2017):

@sqlbot so the appending scenario would work without any problem (assuming the old object is at least 5mb in size, something that in an append scenario seems fairly common).

For random writes it could be still useful in certain limited scenarios (e.g. VM images) where it would still provide substantial benefits.

This was kinda implied by the last phrase in my first post ("This approach would probably be beneficial when the existing object is big and the amount of data to append or changed is small."), I will now update it to clarify it.

@CAFxX commented on GitHub (May 25, 2017): @sqlbot so the appending scenario would work without any problem (assuming the _old_ object is at least 5mb in size, something that in an append scenario seems fairly common). For random writes it could be still useful in certain limited scenarios (e.g. VM images) where it would still provide substantial benefits. This was kinda implied by the last phrase in my first post ("This approach would probably be beneficial when the existing object is big and the amount of data to append or changed is small."), I will now update it to clarify it.

kerem commented

2026-03-04 01:44:36 +03:00

Author

Owner

@ggtakec commented on GitHub (May 2, 2019):

@CAFxX @sqlbot I'm sorry for very late reply.
I implemented this feature in #1027 .
If you can, please try to use master branch latest code after merging it.

@ggtakec commented on GitHub (May 2, 2019): @CAFxX @sqlbot I'm sorry for very late reply. I implemented this feature in #1027 . If you can, please try to use master branch latest code after merging it.

kerem commented

2026-03-04 01:44:37 +03:00

Author

Owner

@ggtakec commented on GitHub (Sep 29, 2019):

This issue is closed because #1098 has been merged.

@ggtakec commented on GitHub (Sep 29, 2019): This issue is closed because #1098 has been merged.