[GH-ISSUE #1941] multipart upload switches to 5MB part size #981

New issue

Open

opened 2026-03-04 01:50:24 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 01:50:24 +03:00

Owner

Originally created by @sbrudenell on GitHub (May 1, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1941

Version of s3fs being used (s3fs --version)

/ # s3fs --version
Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

# apk list fuse
fuse-2.9.9-r1 x86_64 {fuse} (GPL-2.0-only LGPL-2.1-only) [installed]

Kernel information (uname -r)

# uname -r
5.4.43-1-lts

GNU/Linux Distribution, if applicable (cat /etc/os-release)

Using an alpine:edge docker container.

# cat /etc/apk/repositories 
https://dl-cdn.alpinelinux.org/alpine/edge/main
https://dl-cdn.alpinelinux.org/alpine/edge/community
https://dl-cdn.alpinelinux.org/alpine/edge/testing

s3fs command line used, if applicable

s3fs version 1.91(unknown) : s3fs -f -o allow_root -o bucket=sbrudenell-backups -o url=https://s3.us-west-004.backblazeb2.com -o enable_content_md5 -o multipart_size=201 -o use_cache=/tmp -o del_cache -o enable_noobj_cache -o sigv4 -o dbglevel=dbg -o nomixupload /s3

My /tmp/ is a 1gb tmpfs:

Command I used to write a large file:

dd if=/dev/zero of=/var/s3fs/b2.sbrudenell-backups/test.dd bs=1M count=2048 status=progress

Logs illustrating issue

$ docker-compose logs s3fs | grep MultipartUploadRequest | head -n 20
^[[As3fs_1   | 2022-05-01T17:12:56.141Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=0][size=210763776]
s3fs_1   | 2022-05-01T17:13:02.625Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=210763776][size=210763776]
s3fs_1   | 2022-05-01T17:13:09.993Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=421527552][size=210763776]
s3fs_1   | 2022-05-01T17:13:18.120Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=632291328][size=210763776]
s3fs_1   | 2022-05-01T17:13:29.783Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=843055104][size=210763776]
s3fs_1   | 2022-05-01T17:13:47.983Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1053818880][size=19791872]
s3fs_1   | 2022-05-01T17:13:50.126Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1073610752][size=5242880]
s3fs_1   | 2022-05-01T17:13:50.899Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1078853632][size=5242880]
s3fs_1   | 2022-05-01T17:13:51.644Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u0165142
5176107][tpath=/test.dd][fd=9][offset=1084096512][size=5242880]
s3fs_1   | 2022-05-01T17:13:52.365Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1089339392][size=5242880]
s3fs_1   | 2022-05-01T17:13:53.233Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1094582272][size=5242880]
s3fs_1   | 2022-05-01T17:13:53.982Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1099825152][size=5242880]
s3fs_1   | 2022-05-01T17:13:54.707Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1105068032][size=5242880]
s3fs_1   | 2022-05-01T17:13:55.395Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1110310912][size=5242880]
s3fs_1   | 2022-05-01T17:13:56.071Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1115553792][size=5242880]
s3fs_1   | 2022-05-01T17:13:56.814Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1120796672][size=5242880]
s3fs_1   | 2022-05-01T17:13:57.565Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1126039552][size=5242880]
s3fs_1   | 2022-05-01T17:13:58.280Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1131282432][size=5242880]
s3fs_1   | 2022-05-01T17:13:59.062Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1136525312][size=5242880]
s3fs_1   | 2022-05-01T17:13:59.808Z [INF]       curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1141768192][size=5242880]

Details about issue

When writing a large file, s3fs will initially use parts of size multipart_size, but then will start uploading parts of size 5mb. (See the size= arguments in the logs above).

It looks like the uploaded data is correct, but I quickly hit the 10000 part limit due to this issue. I am unable to write any files larger than ~55gb.

I have complete logs. I can post them somewhere secure. I don't want to post them in the issue because it's hard to scan them to make sure I'm not publishing my own secrets.

Tests I've done:

This issue occurs regardless of -o nomixupload. If I do not use -o nomixupload, I see this issue combined with #1936 (large uploads will start with large parts, then use 5mb parts, then "restart" with a new upload_id once the upload reaches 5gb, and again at 10gb, 15gb, etc).
This issue occurs regardless if -o multipart_size is an even divisor of the free cache space, or not. (this is why I used -o multipart_size=201 above)

It seems like something is forcing the use of MIN_MULTIPART_SIZE, which only occurs a few places in the code. I speculate that this is due to:
github.com/s3fs-fuse/s3fs-fuse@a30beded1c/src/fdcache_entity.cpp (L1958)
github.com/s3fs-fuse/s3fs-fuse@a30beded1c/src/fdcache_entity.cpp (L2037)
github.com/s3fs-fuse/s3fs-fuse@a30beded1c/src/fdcache_fdinfo.h (L69)
MIN_MULTIPART_SIZE is being used as a default value here. I'm not sure how s3fs' cache architecture works (if there's good documentation, I've missed it), but this is the best candidate I could find for where this value could even be used.

Originally created by @sbrudenell on GitHub (May 1, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1941 #### Version of s3fs being used (s3fs --version) ``` / # s3fs --version Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` # apk list fuse fuse-2.9.9-r1 x86_64 {fuse} (GPL-2.0-only LGPL-2.1-only) [installed] ``` #### Kernel information (uname -r) ``` # uname -r 5.4.43-1-lts ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) Using an `alpine:edge` docker container. ``` # cat /etc/apk/repositories https://dl-cdn.alpinelinux.org/alpine/edge/main https://dl-cdn.alpinelinux.org/alpine/edge/community https://dl-cdn.alpinelinux.org/alpine/edge/testing ``` #### s3fs command line used, if applicable ``` s3fs version 1.91(unknown) : s3fs -f -o allow_root -o bucket=sbrudenell-backups -o url=https://s3.us-west-004.backblazeb2.com -o enable_content_md5 -o multipart_size=201 -o use_cache=/tmp -o del_cache -o enable_noobj_cache -o sigv4 -o dbglevel=dbg -o nomixupload /s3 ``` My `/tmp/` is a 1gb tmpfs: Command I used to write a large file: ``` dd if=/dev/zero of=/var/s3fs/b2.sbrudenell-backups/test.dd bs=1M count=2048 status=progress ``` #### Logs illustrating issue ``` $ docker-compose logs s3fs | grep MultipartUploadRequest | head -n 20 ^[[As3fs_1 | 2022-05-01T17:12:56.141Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=0][size=210763776] s3fs_1 | 2022-05-01T17:13:02.625Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=210763776][size=210763776] s3fs_1 | 2022-05-01T17:13:09.993Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=421527552][size=210763776] s3fs_1 | 2022-05-01T17:13:18.120Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=632291328][size=210763776] s3fs_1 | 2022-05-01T17:13:29.783Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=843055104][size=210763776] s3fs_1 | 2022-05-01T17:13:47.983Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1053818880][size=19791872] s3fs_1 | 2022-05-01T17:13:50.126Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1073610752][size=5242880] s3fs_1 | 2022-05-01T17:13:50.899Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1078853632][size=5242880] s3fs_1 | 2022-05-01T17:13:51.644Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u0165142 5176107][tpath=/test.dd][fd=9][offset=1084096512][size=5242880] s3fs_1 | 2022-05-01T17:13:52.365Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1089339392][size=5242880] s3fs_1 | 2022-05-01T17:13:53.233Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1094582272][size=5242880] s3fs_1 | 2022-05-01T17:13:53.982Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1099825152][size=5242880] s3fs_1 | 2022-05-01T17:13:54.707Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1105068032][size=5242880] s3fs_1 | 2022-05-01T17:13:55.395Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1110310912][size=5242880] s3fs_1 | 2022-05-01T17:13:56.071Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1115553792][size=5242880] s3fs_1 | 2022-05-01T17:13:56.814Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1120796672][size=5242880] s3fs_1 | 2022-05-01T17:13:57.565Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1126039552][size=5242880] s3fs_1 | 2022-05-01T17:13:58.280Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1131282432][size=5242880] s3fs_1 | 2022-05-01T17:13:59.062Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1136525312][size=5242880] s3fs_1 | 2022-05-01T17:13:59.808Z [INF] curl.cpp:MultipartUploadRequest(4150): [upload_id=4_z1dfeb93105b823ad8705001c_f202026687790bc46_d20220501_m171256_c004_v0402004_t0022_u01651425176107][tpath=/test.dd][fd=9][offset=1141768192][size=5242880] ``` ### Details about issue When writing a large file, s3fs will initially use parts of size `multipart_size`, but then will start uploading parts of size 5mb. (See the `size=` arguments in the logs above). It looks like the uploaded data is correct, but I quickly hit the 10000 part limit due to this issue. I am unable to write any files larger than ~55gb. I have complete logs. I can post them somewhere secure. I don't want to post them in the issue because it's hard to scan them to make sure I'm not publishing my own secrets. Tests I've done: * This issue occurs regardless of `-o nomixupload`. If I do *not* use `-o nomixupload`, I see this issue combined with #1936 (large uploads will start with large parts, then use 5mb parts, then "restart" with a new upload_id once the upload reaches 5gb, and again at 10gb, 15gb, etc). * This issue occurs regardless if `-o multipart_size` is an even divisor of the free cache space, or not. (this is why I used `-o multipart_size=201` above) It seems like something is forcing the use of `MIN_MULTIPART_SIZE`, which only occurs a few places in the code. I speculate that this is due to: https://github.com/s3fs-fuse/s3fs-fuse/blob/a30beded1cd38654c6a5ed8f972a40cd42c144cd/src/fdcache_entity.cpp#L1958 https://github.com/s3fs-fuse/s3fs-fuse/blob/a30beded1cd38654c6a5ed8f972a40cd42c144cd/src/fdcache_entity.cpp#L2037 https://github.com/s3fs-fuse/s3fs-fuse/blob/a30beded1cd38654c6a5ed8f972a40cd42c144cd/src/fdcache_fdinfo.h#L69 `MIN_MULTIPART_SIZE` is being used as a default value here. I'm not sure how s3fs' cache architecture works (if there's good documentation, I've missed it), but this is the best candidate I could find for where this value could even be used.