[GH-ISSUE #1058] Fragmentation parameters do not work when the file size is greater than 8MB and less than 16MB #582

New issue

Closed

opened 2026-03-04 01:46:56 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:46:56 +03:00

Owner

Originally created by @threadfly on GitHub (Jun 30, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1058

Additional Information

S3fs has a multipart_size option, but when the file size is larger than 8MB and less than 16MB, the multipart_size option is specified as 8MB but it does not work. still using normal PUT upload.

Version of s3fs being used (s3fs --version)

s3fs-fuse-1.83
s3fs-fuse-1.85
*) I only compiled these two versions.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

fuse-2.9.2-10.el7.x86_64
fuse-libs-2.9.2-11.el7.x86_64

Kernel information (uname -r)

5.1.5-1.el7.elrepo.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

CentOS Linux 7 (Core)

s3fs command line used, if applicable

s3fs niki-bj /data/test -o url=http://s3.cn-north-1.amazonaws.com.cn -o passwd_file=/data/.passwd-s3fs -o dbglevel=info -o curldbg,use_path_request_style,allow_other -o retries=1 -o multipart_size=8 -o multireq_max=8 -o parallel_count=32

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

Originally created by @threadfly on GitHub (Jun 30, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1058 ### Additional Information S3fs has a multipart_size option, but when the file size is larger than 8MB and less than 16MB, the multipart_size option is specified as 8MB but it does not work. still using normal PUT upload. #### Version of s3fs being used (s3fs --version) s3fs-fuse-1.83 s3fs-fuse-1.85 *) I only compiled these two versions. #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) fuse-2.9.2-10.el7.x86_64 fuse-libs-2.9.2-11.el7.x86_64 #### Kernel information (uname -r) 5.1.5-1.el7.elrepo.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) CentOS Linux 7 (Core) #### s3fs command line used, if applicable ``` s3fs niki-bj /data/test -o url=http://s3.cn-north-1.amazonaws.com.cn -o passwd_file=/data/.passwd-s3fs -o dbglevel=info -o curldbg,use_path_request_style,allow_other -o retries=1 -o multipart_size=8 -o multireq_max=8 -o parallel_count=32 ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue

kerem closed this issue

2026-03-04 01:46:56 +03:00

kerem commented

2026-03-04 01:46:57 +03:00

Author

Owner

@gaul commented on GitHub (Jul 2, 2019):

FdEntity::RowFlush checks to see if the file is double the multipart size before issuing a multipart upload:

    if(pagelist.Size() >= 2 * S3fsCurl::GetMultipartSize() && !nomultipart){ // default 20MB

This is a bit weird but does it pose a practical problem?

@gaul commented on GitHub (Jul 2, 2019): `FdEntity::RowFlush` checks to see if the file is double the multipart size before issuing a multipart upload: ```cpp if(pagelist.Size() >= 2 * S3fsCurl::GetMultipartSize() && !nomultipart){ // default 20MB ``` This is a bit weird but does it pose a practical problem?

kerem commented

2026-03-04 01:46:57 +03:00

Author

Owner

@threadfly commented on GitHub (Jul 3, 2019):

@gaul Some cloud storage implementations of the s3 protocol seem to have restrictions on the size of files uploaded by PUT, so this will cause this problem.

@threadfly commented on GitHub (Jul 3, 2019): @gaul Some cloud storage implementations of the s3 protocol seem to have restrictions on the size of files uploaded by PUT, so this will cause this problem.

kerem referenced this issue

2026-03-04 02:01:02 +03:00

[PR #582] [MERGED] Check errors returned in 200 OK responses for put header request #1556