[GH-ISSUE #941] automatically tune multipart sizes #532

New issue

Open

opened 2026-03-04 01:46:26 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:46:26 +03:00

Owner

Originally created by @gaul on GitHub (Jan 30, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/941

s3fs should automatically use larger multipart sizes when object sizes are large. For example, multipart_size defaults to 10 MB which means that s3fs can only write <= 100 GB objects with the maximum 10,000 part size instead of the 5 TB limit. Similarly, singlepart_copy_limit should start smaller to improve parallel uploads but increase as object size gets larger. Propose giving these -1 defaults to allow users to modify behavior but otherwise letting s3fs choose the sizes. References #940.

Originally created by @gaul on GitHub (Jan 30, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/941 s3fs should automatically use larger multipart sizes when object sizes are large. For example, `multipart_size` defaults to 10 MB which means that s3fs can only write <= 100 GB objects with the maximum 10,000 part size instead of the 5 TB limit. Similarly, `singlepart_copy_limit` should start smaller to improve parallel uploads but increase as object size gets larger. Propose giving these `-1` defaults to allow users to modify behavior but otherwise letting s3fs choose the sizes. References #940.

kerem added the

enhancement

label

2026-03-04 01:46:26 +03:00

kerem commented

2026-03-04 01:46:27 +03:00

Author

Owner

@ffeldhaus commented on GitHub (Feb 1, 2019):

I would suggest to divide the filesize by the parallel_count or a multiple of parallel_count and determine the multipart_size that way. It's also helpful if the multipart_size is rounded down to the next power of 2 (e.g. 16MB or 1GB) in case someone wants to check the ETag of a downloaded file and needs to guess the part size used for the multipart upload.

@ffeldhaus commented on GitHub (Feb 1, 2019): I would suggest to divide the filesize by the `parallel_count` or a multiple of `parallel_count` and determine the `multipart_size` that way. It's also helpful if the `multipart_size` is rounded down to the next power of 2 (e.g. 16MB or 1GB) in case someone wants to check the ETag of a downloaded file and needs to guess the part size used for the multipart upload.

kerem commented

2026-03-04 01:46:27 +03:00

Author

Owner

@gaul commented on GitHub (Feb 2, 2019):

Using more parts than the number of parallel_count helps since network errors do not retransmit as much data. The underlying curl reuses connections so TCP window scaling only affects some connections. Note that there are many possible values for part sizes; let's scope this one to simply allow the full range of object sizes and we can follow on with optimizations.

@gaul commented on GitHub (Feb 2, 2019): Using more parts than the number of `parallel_count` helps since network errors do not retransmit as much data. The underlying curl reuses connections so TCP window scaling only affects some connections. Note that there are many possible values for part sizes; let's scope this one to simply allow the full range of object sizes and we can follow on with optimizations.