[GH-ISSUE #1556] Consider reducing multipart_copy_size #815

New issue

Open

opened 2026-03-04 01:49:02 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:49:02 +03:00

Owner

Originally created by @gaul on GitHub (Feb 8, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1556

s3fs can unlock more performance by reducing the default value for multipart_copy_size from 512 MB. S3 is implemented as a distributed system and more parallel copies with smaller part sizes can improve mixupload, rename, and chmod performance. Need to experiment with some different sizes to find a good value. One downside is that the current default of 512 MB can support the maximum object size of 5 TB when using 10,000 parts. If we reduce the value, users will need to override it via #1555.

Originally created by @gaul on GitHub (Feb 8, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1556 s3fs can unlock more performance by reducing the default value for `multipart_copy_size` from 512 MB. S3 is implemented as a distributed system and more parallel copies with smaller part sizes can improve mixupload, rename, and chmod performance. Need to experiment with some different sizes to find a good value. One downside is that the current default of 512 MB can support the maximum object size of 5 TB when using 10,000 parts. If we reduce the value, users will need to override it via #1555.

kerem added the

performance

help wanted

labels

2026-03-04 01:49:02 +03:00

kerem commented

2026-03-04 01:49:02 +03:00

Author

Owner

@gaul commented on GitHub (Jul 31, 2021):

@CarstenGrohmann I could use some help with tuning this value. I believe that copies as small as 20 MB (2 * 10 MB copies) might benefit from parallelism. However it needs some benchmarking to prove that a value smaller than 512 MB is useful, both with LAN- and WAN-access. Any interest in investigating?

@gaul commented on GitHub (Jul 31, 2021): @CarstenGrohmann I could use some help with tuning this value. I believe that copies as small as 20 MB (2 * 10 MB copies) might benefit from parallelism. However it needs some benchmarking to prove that a value smaller than 512 MB is useful, both with LAN- and WAN-access. Any interest in investigating?

kerem commented

2026-03-04 01:49:02 +03:00

Author

Owner

@CarstenGrohmann commented on GitHub (Aug 1, 2021):

Yes, even though I will probably only have access to a local S3 appliance for another 6 to 8 weeks.

If the results of these tests differ too much, then you might be pushed to introduce profiles that set the values together for a scenario (AWS S3 over WAN, AWS Outpost, MinIO over LAN, NetApp StorageGrid over LAN).

@CarstenGrohmann commented on GitHub (Aug 1, 2021): Yes, even though I will probably only have access to a local S3 appliance for another 6 to 8 weeks. If the results of these tests differ too much, then you might be pushed to introduce profiles that set the values together for a scenario (AWS S3 over WAN, AWS Outpost, MinIO over LAN, NetApp StorageGrid over LAN).