[GH-ISSUE #1980] File is downloading in 24MB chunks even with -o multipart_size=5 #1000

New issue

Open

opened 2026-03-04 01:50:33 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:50:33 +03:00

Owner

Originally created by @akskos on GitHub (Jul 8, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1980

Additional Information

I'm trying to stream a video into a video player from s3fs and tried to set multipart_size=5 to make it start faster. However the video player only gets any data when the first 24MB have been downloaded. I also tried this by copying a file from s3fs to my local fs with dd setting block size to 1MB and i'm getting the first chunk of data at 24MB.

Version of s3fs being used (s3fs --version)

V1.90

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.9

Kernel information (uname -r)

5.14.18-100.fc33.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME=Fedora
VERSION="33 (Workstation Edition)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Workstation Edition"
VARIANT_ID=workstation

s3fs command line used, if applicable

s3fs bucket.name mount-point -o passwd_file=./.passwd-s3fs -o use_path_request_style -o endpoint=eu-north-1 -o url=http://s3-eu-north-1.amazonaws.com/ -o multipart_size=5 -o multipart_copy_size=5

Originally created by @akskos on GitHub (Jul 8, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1980 ### Additional Information I'm trying to stream a video into a video player from s3fs and tried to set multipart_size=5 to make it start faster. However the video player only gets any data when the first 24MB have been downloaded. I also tried this by copying a file from s3fs to my local fs with dd setting block size to 1MB and i'm getting the first chunk of data at 24MB. #### Version of s3fs being used (s3fs --version) V1.90 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.9 #### Kernel information (uname -r) 5.14.18-100.fc33.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME=Fedora VERSION="33 (Workstation Edition)" ID=fedora VERSION_ID=33 VERSION_CODENAME="" PLATFORM_ID="platform:f33" PRETTY_NAME="Fedora 33 (Workstation Edition)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:33" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/" SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=33 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=33 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Workstation Edition" VARIANT_ID=workstation #### s3fs command line used, if applicable ``` s3fs bucket.name mount-point -o passwd_file=./.passwd-s3fs -o use_path_request_style -o endpoint=eu-north-1 -o url=http://s3-eu-north-1.amazonaws.com/ -o multipart_size=5 -o multipart_copy_size=5 ```

kerem commented

2026-03-04 01:50:34 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 9, 2022):

Currently s3fs tries to download the size specified by multipart_size and parallel_count(ex, if both parameters are set 5, download size is 25MB).
This is so that the download cache can be stored effectively.
I think this is the cause of the delay until the start of reading.

The only way to deal with this is to reduce the number of parallels at the time of download, but I think that the performance after the start will be worse.
This is a difficult problem for each application on case.

@gaul Please give us your opinion.

@ggtakec commented on GitHub (Jul 9, 2022): Currently s3fs tries to download the size specified by multipart_size and parallel_count(ex, if both parameters are set 5, download size is 25MB). This is so that the download cache can be stored effectively. I think this is the cause of the delay until the start of reading. The only way to deal with this is to reduce the number of parallels at the time of download, but I think that the performance after the start will be worse. This is a difficult problem for each application on case. @gaul Please give us your opinion.