starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #816] Can you force every file upload operation to be multi-part? #472

New issue

Closed

opened 2026-03-04 01:45:55 +03:00 by kerem · 6 comments

kerem commented

2026-03-04 01:45:55 +03:00

Owner

Originally created by @blidarescub on GitHub (Sep 4, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/816

I have a scenario where I need every file to be uploaded in multi-part, because it triggers some events in SQS. Currently, only files bigger than 10MB are being uploaded in multi-part. I could not find any combination of command arguments that changes this.
Thanks

Originally created by @blidarescub on GitHub (Sep 4, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/816 I have a scenario where I need every file to be uploaded in multi-part, because it triggers some events in SQS. Currently, only files bigger than 10MB are being uploaded in multi-part. I could not find any combination of command arguments that changes this. Thanks

kerem closed this issue

2026-03-04 01:45:55 +03:00

kerem commented

2026-03-04 01:45:56 +03:00

Author

Owner

@gaul commented on GitHub (Oct 19, 2018):

Could you set -o multipart_size to zero and see if this works? Otherwise we might need some tweaks on this code path and we would accept PRs for this.

@gaul commented on GitHub (Oct 19, 2018): Could you set `-o multipart_size` to zero and see if this works? Otherwise we might need some tweaks on this code path and we would accept PRs for this.

kerem commented

2026-03-04 01:45:56 +03:00

Author

Owner

@kahing commented on GitHub (Oct 19, 2018):

Files smaller than 5MB cannot use multipart upload

@kahing commented on GitHub (Oct 19, 2018): Files smaller than 5MB cannot use multipart upload

kerem commented

2026-03-04 01:45:56 +03:00

Author

Owner

@gaul commented on GitHub (Oct 19, 2018):

According to s3-tests, 1 byte MPUs work but 0 byte fail with MalformedXML.

@gaul commented on GitHub (Oct 19, 2018): According to s3-tests, 1 byte MPUs work but 0 byte fail with `MalformedXML`.

kerem commented

2026-03-04 01:45:57 +03:00

Author

Owner

@gaul commented on GitHub (Jan 24, 2019):

@blidarescub Did my -o multipart_size suggestion address your request?

@gaul commented on GitHub (Jan 24, 2019): @blidarescub Did my `-o multipart_size` suggestion address your request?

kerem commented

2026-03-04 01:45:57 +03:00

Author

Owner

@blidarescub commented on GitHub (Jan 24, 2019):

@gaul As far as i remember it did not, the winning combo for me was this: -o use_cache=/tmp -o allow_other -o uid=1000 -o singlepart_copy_limit=5 -o multipart_size=5 -o multireq_max=5

@blidarescub commented on GitHub (Jan 24, 2019): @gaul As far as i remember it did not, the winning combo for me was this: `-o use_cache=/tmp -o allow_other -o uid=1000 -o singlepart_copy_limit=5 -o multipart_size=5 -o multireq_max=5`

kerem commented

2026-03-04 01:45:57 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 29, 2019):

s3fs internally has the lower limit of the multipart_size option.
Its value is 5, which means 5MB.
If it is necessary to be able to set a value smaller than this, we will be able to allow to exclude the lower limit.
However, even if it is possible to set a minimum value of 1 byte, it seems that s3fs creates a 0 byte file before first file upload, so it may not be possible to avoid the SQS action.

@ggtakec commented on GitHub (Mar 29, 2019): s3fs internally has the lower limit of the multipart_size option. Its value is `5`, which means `5MB`. If it is necessary to be able to set a value smaller than this, we will be able to allow to exclude the lower limit. However, even if it is possible to set a minimum value of 1 byte, it seems that s3fs creates a 0 byte file before first file upload, so it may not be possible to avoid the SQS action.