[GH-ISSUE #2068] large file upload faild #1044

New issue

Closed

opened 2026-03-04 01:50:54 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:50:54 +03:00

Owner

Originally created by @welyss on GitHub (Nov 29, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2068

Version of s3fs being used (`s3fs --version`)

Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

2.9.2

Kernel information (`uname -r`)

3.10.0-1062.el7.x86_64

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

CentOS Linux release 7.7.1908 (Core)

How to run s3fs, if applicable

[dbsbackup /wystest fuse.s3fs _netdev,allow_other,use_path_request_style,url=http://thor.dev.hbec.com/namespaces/rook-ceph/services/rook-ceph-rgw-hbec-store,uid=1000,gid=1001,logfile=/root/s3fs.log,dbglevel=info,curldbg 0 0
] /etc/fstab

dd if=/dev/zero of=wystestdata bs=1024 count=6144000 status=progress

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

2022-11-29T06:48:09.943Z [CURL DBG] > PUT /namespaces/rook-ceph/services/rook-ceph-rgw-hbec-store/dbsbackup/abc/wystestdata?partNumber=6&uploadId=2~AO9TzkLf1tDeY7MZZOOkAbBSGDFJdac HTTP/1.1
2022-11-29T06:48:09.943Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash unknown; OpenSSL)
2022-11-29T06:48:09.943Z [CURL DBG] > Accept: */*
2022-11-29T06:48:09.943Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=H3UNWD369B2RGE62SO3R/20221129/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-range;x-amz-date, Signature=0fb7086d546fe0eb10420e49e38c5d3ec70e72495b60aca2fd44e30998845ec4
2022-11-29T06:48:09.943Z [CURL DBG] > Content-Type: application/octet-stream
2022-11-29T06:48:09.943Z [CURL DBG] > host: thor.dev.hbec.com
2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-copy-source: /dbsbackup/abc/wystestdata
2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-copy-source-range: bytes=2684354560-3221225471
2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-date: 20221129T064809Z
2022-11-29T06:48:09.943Z [CURL DBG] > Content-Length: 0
2022-11-29T06:48:09.943Z [CURL DBG] > Expect: 100-continue
2022-11-29T06:48:09.943Z [CURL DBG] >
2022-11-29T06:48:09.945Z [CURL DBG] < HTTP/1.1 100 Continue
2022-11-29T06:49:15.128Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.128Z [CURL DBG] * Callback aborted

2022-11-29T06:49:15.128Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.128Z [CURL DBG] * Callback aborted

2022-11-29T06:49:15.128Z [CURL DBG] * Closing connection 6

2022-11-29T06:49:15.128Z [ERR] curl.cpp:RequestPerform(2434): ### CURLE_ABORTED_BY_CALLBACK
2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted

2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted

2022-11-29T06:49:15.129Z [CURL DBG] * Closing connection 9

2022-11-29T06:49:15.129Z [ERR] curl.cpp:RequestPerform(2434): ### CURLE_ABORTED_BY_CALLBACK
2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted

2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted

2022-11-29T06:49:15.129Z [CURL DBG] * Closing connection 7

2022-11-29T06:49:15.129Z [ERR] curl.cpp:RequestPerform(2434): ### CURLE_ABORTED_BY_CALLBACK
2022-11-29T06:49:15.137Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120
2022-11-29T06:49:15.137Z [CURL DBG] * Callback aborted

https://github.com/welyss/mypubinfo/blob/38758f375da661ac434fa81fa64a6b9abeac101d/s3fs_bak.log

Details about issue

the file which size over max_dirty_data (default="5120"), my case is 6G(4G success), after progress stay near in 5G, CopyMultipartPost would be called, for now everything is ok, when 5G done, the file which left 2G start upload, error throw, and process never done, holding on front

Originally created by @welyss on GitHub (Nov 29, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2068 #### Version of s3fs being used (`s3fs --version`) Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 2.9.2 #### Kernel information (`uname -r`) 3.10.0-1062.el7.x86_64 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) CentOS Linux release 7.7.1908 (Core) #### How to run s3fs, if applicable  [dbsbackup /wystest fuse.s3fs _netdev,allow_other,use_path_request_style,url=http://thor.dev.hbec.com/namespaces/rook-ceph/services/rook-ceph-rgw-hbec-store,uid=1000,gid=1001,logfile=/root/s3fs.log,dbglevel=info,curldbg 0 0 ] /etc/fstab ``` dd if=/dev/zero of=wystestdata bs=1024 count=6144000 status=progress ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` 2022-11-29T06:48:09.943Z [CURL DBG] > PUT /namespaces/rook-ceph/services/rook-ceph-rgw-hbec-store/dbsbackup/abc/wystestdata?partNumber=6&uploadId=2~AO9TzkLf1tDeY7MZZOOkAbBSGDFJdac HTTP/1.1 2022-11-29T06:48:09.943Z [CURL DBG] > User-Agent: s3fs/1.91 (commit hash unknown; OpenSSL) 2022-11-29T06:48:09.943Z [CURL DBG] > Accept: */* 2022-11-29T06:48:09.943Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=H3UNWD369B2RGE62SO3R/20221129/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-range;x-amz-date, Signature=0fb7086d546fe0eb10420e49e38c5d3ec70e72495b60aca2fd44e30998845ec4 2022-11-29T06:48:09.943Z [CURL DBG] > Content-Type: application/octet-stream 2022-11-29T06:48:09.943Z [CURL DBG] > host: thor.dev.hbec.com 2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-copy-source: /dbsbackup/abc/wystestdata 2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-copy-source-range: bytes=2684354560-3221225471 2022-11-29T06:48:09.943Z [CURL DBG] > x-amz-date: 20221129T064809Z 2022-11-29T06:48:09.943Z [CURL DBG] > Content-Length: 0 2022-11-29T06:48:09.943Z [CURL DBG] > Expect: 100-continue 2022-11-29T06:48:09.943Z [CURL DBG] > 2022-11-29T06:48:09.945Z [CURL DBG] < HTTP/1.1 100 Continue 2022-11-29T06:49:15.128Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.128Z [CURL DBG] * Callback aborted 2022-11-29T06:49:15.128Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.128Z [CURL DBG] * Callback aborted 2022-11-29T06:49:15.128Z [CURL DBG] * Closing connection 6 2022-11-29T06:49:15.128Z [ERR] curl.cpp:RequestPerform(2434): ### CURLE_ABORTED_BY_CALLBACK 2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted 2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted 2022-11-29T06:49:15.129Z [CURL DBG] * Closing connection 9 2022-11-29T06:49:15.129Z [ERR] curl.cpp:RequestPerform(2434): ### CURLE_ABORTED_BY_CALLBACK 2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted 2022-11-29T06:49:15.129Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.129Z [CURL DBG] * Callback aborted 2022-11-29T06:49:15.129Z [CURL DBG] * Closing connection 7 2022-11-29T06:49:15.129Z [ERR] curl.cpp:RequestPerform(2434): ### CURLE_ABORTED_BY_CALLBACK 2022-11-29T06:49:15.137Z [ERR] curl.cpp:CurlProgress(356): timeout now: 1669704555, curl_times[curl]: 1669704434, readwrite_timeout: 120 2022-11-29T06:49:15.137Z [CURL DBG] * Callback aborted ``` [https://github.com/welyss/mypubinfo/blob/38758f375da661ac434fa81fa64a6b9abeac101d/s3fs_bak.log](url) ### Details about issue the file which size over max_dirty_data (default="5120"), my case is 6G(4G success), after progress stay near in 5G, CopyMultipartPost would be called, for now everything is ok, when 5G done, the file which left 2G start upload, error throw, and process never done, holding on front