starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #324] Multipart Upload fails with 411 Length Required #168

New issue

Closed

opened 2026-03-04 01:42:48 +03:00 by kerem · 8 comments

kerem commented

2026-03-04 01:42:48 +03:00

Owner

Originally created by @anaelcarvalho on GitHub (Jan 8, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/324

When copying a file that requires a multipart upload (file size greater than 10MB by default), the Initiate Multipart Upload call fails with a status code of 411 Length Required. As per AWS API documentation, this status code means that the "Content-Length" header is missing. Curl debug information shows that this header was not sent:

POST http://.s3.amazonaws.com/testfile.zip?uploads= HTTP/1.1
Proxy-Connection: Keep-Alive
Authorization: <AUTH_HEADER>
Content-Type: application/zip
host: .s3.amazonaws.com
x-amz-acl: private
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20160108T172803Z
x-amz-meta-gid: 0
x-amz-meta-mode: 33188
x-amz-meta-mtime: 1452274083
x-amz-meta-uid: 0

Changing the following line in curl.cpp

2835     requestHeaders = curl_slist_sort_insert(requestHeaders, "Content-Length", NULL);

To the below:

2835     requestHeaders = curl_slist_sort_insert(requestHeaders, "Content-Length", "0");

Appears to have fixed the issue, with "Content-Length" now being sent with a value of 0:

POST http://.s3.amazonaws.com/testfile.zip?uploads= HTTP/1.1
Proxy-Connection: Keep-Alive
Authorization: <AUTH_HEADER>
Content-Length: 0
Content-Type: application/zip
host: .s3.amazonaws.com
x-amz-acl: private
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date: 20160108T185612Z
x-amz-meta-gid: 0
x-amz-meta-mode: 33188
x-amz-meta-mtime: 1452279372
x-amz-meta-uid: 0

This is a bit confusing, though, as the documentation doesn't list this header as required (in fact, the sample requests do not even include it). Should it be treated as a bug?

Thanks!

Originally created by @anaelcarvalho on GitHub (Jan 8, 2016). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/324 When copying a file that requires a multipart upload (file size greater than 10MB by default), the [Initiate Multipart Upload](http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html) call fails with a status code of 411 Length Required. As per [AWS API documentation](http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html), this status code means that the "Content-Length" header is missing. Curl debug information shows that this header was not sent: > POST http://<BUCKET>.s3.amazonaws.com/testfile.zip?uploads= HTTP/1.1 > Proxy-Connection: Keep-Alive > Authorization: <AUTH_HEADER> > Content-Type: application/zip > host: <BUCKET>.s3.amazonaws.com > x-amz-acl: private > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 > x-amz-date: 20160108T172803Z > x-amz-meta-gid: 0 > x-amz-meta-mode: 33188 > x-amz-meta-mtime: 1452274083 > x-amz-meta-uid: 0 Changing the following line in **curl.cpp** ``` cpp 2835 requestHeaders = curl_slist_sort_insert(requestHeaders, "Content-Length", NULL); ``` To the below: ``` cpp 2835 requestHeaders = curl_slist_sort_insert(requestHeaders, "Content-Length", "0"); ``` Appears to have fixed the issue, with "Content-Length" now being sent with a value of 0: > POST http://<BUCKET>.s3.amazonaws.com/testfile.zip?uploads= HTTP/1.1 > Proxy-Connection: Keep-Alive > Authorization: <AUTH_HEADER> > Content-Length: 0 > Content-Type: application/zip > host: <BUCKET>.s3.amazonaws.com > x-amz-acl: private > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 > x-amz-date: 20160108T185612Z > x-amz-meta-gid: 0 > x-amz-meta-mode: 33188 > x-amz-meta-mtime: 1452279372 > x-amz-meta-uid: 0 This is a bit confusing, though, as the [documentation](http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadInitiate.html) doesn't list this header as required (in fact, the sample requests do not even include it). Should it be treated as a bug? Thanks!

kerem closed this issue

2026-03-04 01:42:49 +03:00

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@gaul commented on GitHub (Jan 9, 2016):

@anaelcarvalho Could you provide the exact steps to reproduce your symptoms as well as which version of s3fs you use?

@gaul commented on GitHub (Jan 9, 2016): @anaelcarvalho Could you provide the exact steps to reproduce your symptoms as well as which version of s3fs you use?

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@anaelcarvalho commented on GitHub (Jan 9, 2016):

Sure -

Mount bucket to mount point
s3fs <BUCKET> /mnt/mountpoint -o passwd_file=/path/to/passwd
Copy test file (47 MB file) to mount point
cp testfile.zip /mnt/mountpoint

Built from master branch, commit ea151a7 - don't think it has a version tag yet (1.80-dev?)

@anaelcarvalho commented on GitHub (Jan 9, 2016): Sure - 1. Mount bucket to mount point `s3fs <BUCKET> /mnt/mountpoint -o passwd_file=/path/to/passwd` 2. Copy test file (47 MB file) to mount point `cp testfile.zip /mnt/mountpoint` Built from master branch, commit ea151a7 - don't think it has a version tag yet (1.80-dev?)

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 9, 2016):

@anaelcarvalho
I tried multipart upload with 32-64MB files, but could not get same 411 error.
I uploaded files by sigv4 and sigv2, both are succeed without error.

I have some question about your environment.
Do you use any proxy to s3.amazonaws.com?
And we should know about the region on your bucket.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 9, 2016): @anaelcarvalho I tried multipart upload with 32-64MB files, but could not get same 411 error. I uploaded files by sigv4 and sigv2, both are succeed without error. I have some question about your environment. Do you use any proxy to s3.amazonaws.com? And we should know about the region on your bucket. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@anaelcarvalho commented on GitHub (Jan 9, 2016):

@ggtakec
I tried sigv2 in addition to the default sigv4 with same results.

No proxy, region is sa-east-1. I suspect this region may have a slightly different configuration that requires the "Content-Length" header parameter. Haven't tried a different region yet, though.

@anaelcarvalho commented on GitHub (Jan 9, 2016): @ggtakec I tried sigv2 in addition to the default sigv4 with same results. No proxy, region is **sa-east-1**. I suspect this region may have a slightly different configuration that requires the "Content-Length" header parameter. Haven't tried a different region yet, though.

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 9, 2016):

@anaelcarvalho Thanks a lot.

If you can, please try to run s3fs with "dbglevel" and "curldbg"(please see man page for detail).
It puts many debug messages, I want to know Initial-MultipartUpload request and response which are put by s3fs.

I don't know about "Proxy-Connection: Keep-Alive" in your first curl logging message, who adds this header.
So only I could not find this much difference, please try to run with debug option and check log.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 9, 2016): @anaelcarvalho Thanks a lot. If you can, please try to run s3fs with "dbglevel" and "curldbg"(please see man page for detail). It puts many debug messages, I want to know Initial-MultipartUpload request and response which are put by s3fs. I don't know about "Proxy-Connection: Keep-Alive" in your first curl logging message, who adds this header. So only I could not find this much difference, please try to run with debug option and check log. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@anaelcarvalho commented on GitHub (Jan 11, 2016):

@ggtakec

After debugging further, I found that your suspicion is correct - the connection is in fact being routed through a proxy due to unavailability of the VPC Endpoint in this region (SA).

Turns out that the proxy being used (Squid) requires the 'Content-Length' header parameter in POST requests until version 3.1.10; this requirement being dropped from 3.1.11 onwards.

We will upgrade the proxy version and retest, but I expect that this should do it.

Thank you!

@anaelcarvalho commented on GitHub (Jan 11, 2016): @ggtakec After debugging further, I found that your suspicion is correct - the connection is in fact being routed through a proxy due to unavailability of the [VPC Endpoint](https://aws.amazon.com/pt/blogs/aws/new-vpc-endpoint-for-amazon-s3/) in this region (SA). Turns out that the proxy being used (Squid) [requires the 'Content-Length' header parameter in POST requests until version 3.1.10](http://www.squid-cache.org/mail-archive/squid-users/201101/0280.html); this [requirement being dropped from 3.1.11 onwards](http://bazaar.launchpad.net/~squid/squid/3.1/revision/10214). We will upgrade the proxy version and retest, but I expect that this should do it. Thank you!

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@anaelcarvalho commented on GitHub (Jan 11, 2016):

Retested with new proxy and fixed. Closing issue.

@anaelcarvalho commented on GitHub (Jan 11, 2016): Retested with new proxy and fixed. Closing issue.

kerem commented

2026-03-04 01:42:50 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 16, 2016):