[GH-ISSUE #1457] Authorization header missing when retrying request #767

Closed
opened 2026-03-04 01:48:36 +03:00 by kerem · 6 comments
Owner

Originally created by @ssvinarev on GitHub (Oct 20, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1457

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

1.87

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2-11

Kernel information (uname -r)

3.10.0-1127.19.1.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

CentOS Linux release 7.8.2003 (Core)

s3fs command line used, if applicable

s3fs -d backup-cold-bucket-1:/projectname /mnt/data1 -o url=https://storage.yandexcloud.net -o endpoint=ru-central1 -o use_path_request_style -o storage_class=standard_ia -o notsup_compat_dir -o max_stat_cache_size=1000000 -o parallel_count=20 -o multipart_size=99 -o singlepart_copy_limit=-1 -o nomixupload -o use_cache=/mnt/cache -o retries=15 -o connect_timeout=60 -o readwrite_timeout=60 -o curldbg -o allow_other,uid=997,gid=99

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Oct  7 12:17:54 hostname s3fs[6208]: * Connection #2091 to host storage.yandexcloud.net left intact
Oct  7 12:17:54 hostname s3fs[6208]:      HTTP response code 200
Oct  7 12:17:57 hostname s3fs[6208]:      [upload_id=0005B10F73667DBC][tpath=/stor1-data-full-odd-0301][fd=50][offset=110820259585][size=104896510]
Oct  7 12:17:57 hostname s3fs[6208]:      [tpath=/stor1-data-full-odd-0301][start=110820259585][size=104896510][part=1058]
Oct  7 12:17:57 hostname s3fs[6208]:      [tpath=/stor1-data-full-odd-0301][start=110820259585][size=104896510][part=1058]
Oct  7 12:17:57 hostname s3fs[6208]:      URL is https://storage.yandexcloud.net/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DB
C
Oct  7 12:17:57 hostname s3fs[6208]:      URL changed is https://storage.yandexcloud.net/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10
F73667DBC
Oct  7 12:17:58 hostname s3fs[6208]:      computing signature [PUT] [/projectname/stor1-data-full-odd-0301] [partNumber=1058&uploadId=0005B10F73667DBC] [1764f1ee9310ab8495a335181c4503
61f3f6cabe1fe1ad6c8645143e5681398e]
Oct  7 12:17:58 hostname s3fs[6208]:      url is https://storage.yandexcloud.net
Oct  7 12:17:58 hostname s3fs[6208]: * Found bundle for host storage.yandexcloud.net: 0x7fef1c0012e0 [serially]
Oct  7 12:17:58 hostname s3fs[6208]: * Can not multiplex, even if we wanted to!
Oct  7 12:17:58 hostname s3fs[6208]: * Re-using existing connection! (#2091) with host storage.yandexcloud.net
Oct  7 12:17:58 hostname s3fs[6208]: * Connected to storage.yandexcloud.net (213.180.193.243) port 443 (#2091)
Oct  7 12:17:58 hostname s3fs[6208]: > PUT /backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DBC HTTP/1.1
Oct  7 12:17:58 hostname s3fs[6208]: > Host: storage.yandexcloud.net
Oct  7 12:17:58 hostname s3fs[6208]: > User-Agent: s3fs/1.87 (commit hash unknown; OpenSSL)
Oct  7 12:17:58 hostname s3fs[6208]: > Authorization: AWS4-HMAC-SHA256 Credential=zxcRez4ooKDUB-KMGYQb/20201007/ru-central1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-
date, Signature=983a8bfebf7fe92af217af987840e23555a9c997b744feaac7f73955520f07e3
Oct  7 12:17:58 hostname s3fs[6208]: > x-amz-content-sha256: 1764f1ee9310ab8495a335181c450361f3f6cabe1fe1ad6c8645143e5681398e
Oct  7 12:17:58 hostname s3fs[6208]: > x-amz-date: 20201007T091758Z
Oct  7 12:17:58 hostname s3fs[6208]: > Content-Length: 104896510
Oct  7 12:17:58 hostname s3fs[6208]: > Expect: 100-continue
Oct  7 12:17:58 hostname s3fs[6208]: >
Oct  7 12:17:59 hostname s3fs[6208]: * Done waiting for 100-continue
Oct  7 12:23:00 hostname s3fs[6208]: curl.cpp:CurlProgress(662): timeout now: 1602062580, curl_times[curl]: 1602062279, readwrite_timeout: 300
Oct  7 12:23:00 hostname s3fs[6208]: * Callback aborted
Oct  7 12:23:00 hostname s3fs[6208]: * Closing connection 2091
Oct  7 12:23:00 hostname s3fs[6208]: curl.cpp:RequestPerform(2601): ### CURLE_ABORTED_BY_CALLBACK
Oct  7 12:23:04 hostname s3fs[6208]: ### retrying...
Oct  7 12:23:04 hostname s3fs[6208]:      Retry request. [type=9][url=https://storage.yandexcloud.net/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DBC][path=/projectname/stor1-data-full-odd-0301]
Oct  7 12:23:04 hostname s3fs[6208]: *   Trying 213.180.193.243:443...
Oct  7 12:23:04 hostname s3fs[6208]: * Connected to storage.yandexcloud.net (213.180.193.243) port 443 (#2092)
Oct  7 12:23:04 hostname s3fs[6208]: *   CAfile: none#012  CApath: none
Oct  7 12:23:04 hostname s3fs[6208]: * ALPN/NPN, server did not agree to a protocol
Oct  7 12:23:04 hostname s3fs[6208]: * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Oct  7 12:23:04 hostname s3fs[6208]: * Server certificate:
Oct  7 12:23:04 hostname s3fs[6208]: *     subject: CN=*.website.yandexcloud.net,O=Yandex LLC,OU=ITO,L=Moscow,C=RU
Oct  7 12:23:04 hostname s3fs[6208]: *     start date: Oct 02 09:31:15 2020 GMT
Oct  7 12:23:04 hostname s3fs[6208]: *     expire date: Apr 02 09:31:15 2021 GMT
Oct  7 12:23:04 hostname s3fs[6208]: *     common name: *.website.yandexcloud.net
Oct  7 12:23:04 hostname s3fs[6208]: *     issuer: CN=Yandex CA,OU=Yandex Certification Authority,O=Yandex LLC,C=RU
Oct  7 12:23:04 hostname s3fs[6208]: > PUT /backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DBC HTTP/1.1
Oct  7 12:23:04 hostname s3fs[6208]: > Host: storage.yandexcloud.net
Oct  7 12:23:04 hostname s3fs[6208]: > User-Agent: s3fs/1.87 (commit hash unknown; OpenSSL)
Oct  7 12:23:04 hostname s3fs[6208]: > Accept: */*
Oct  7 12:23:04 hostname s3fs[6208]: > Content-Length: 104896510
Oct  7 12:23:04 hostname s3fs[6208]: > Expect: 100-continue
Oct  7 12:23:04 hostname s3fs[6208]: >
Oct  7 12:23:04 hostname s3fs[6208]: * Mark bundle as not supporting multiuse
Oct  7 12:23:04 hostname s3fs[6208]: < HTTP/1.1 100 Continue
Oct  7 12:23:04 hostname s3fs[6208]: * Mark bundle as not supporting multiuse
Oct  7 12:23:04 hostname s3fs[6208]: < HTTP/1.1 403 Forbidden
Oct  7 12:23:04 hostname s3fs[6208]: < Server: nginx
Oct  7 12:23:04 hostname s3fs[6208]: < Date: Wed, 07 Oct 2020 09:23:04 GMT
Oct  7 12:23:04 hostname s3fs[6208]: < Content-Type: application/xml; charset=UTF-8
Oct  7 12:23:04 hostname s3fs[6208]: < Content-Length: 283
Oct  7 12:23:04 hostname s3fs[6208]: < Connection: keep-alive
Oct  7 12:23:04 hostname s3fs[6208]: < Keep-Alive: timeout=60
Oct  7 12:23:04 hostname s3fs[6208]: < X-Amz-Request-Id: 55f12ff942d080f5
Oct  7 12:23:04 hostname s3fs[6208]: * HTTP error before end of send, keep sending
Oct  7 12:23:04 hostname s3fs[6208]: <
Oct  7 12:23:05 hostname s3fs[6208]: * We are completely uploaded and fine
Oct  7 12:23:05 hostname s3fs[6208]: * Connection #2092 to host storage.yandexcloud.net left intact
Oct  7 12:23:05 hostname s3fs[6208]: curl.cpp:RequestPerform(2540): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&amp;uploadId=0005B10F73667DBC</Resource><RequestId>55f12ff942d080f5</RequestId></Error>
Oct  7 12:23:05 hostname s3fs[6208]: curl.cpp:MultipartUploadRequest(4224): failed uploading part(-1)
Oct  7 12:23:05 hostname s3fs[6208]: fdcache.cpp:Write(2562): failed to multipart post(start=110820259585, size=104896510) for file(50).
Oct  7 12:23:05 hostname s3fs[6208]: s3fs.cpp:s3fs_write(2240): failed to write file(/stor1-data-full-odd-0301). result=-1

Details about issue

Hello!
Maybe someone can help in troubleshooting.

In this log example there is attempt to upload partNumber=1058 of an object "stor1-data-full-odd-0301". This PUT-request has "Authorization" header. Fine!
Request failed due to timeout.
Then closing connection 2091 and retrying with a new connection 2092. Fine to!
But next PUT-request doesn't have "Authorization" header and so gets response HTTP/1.1 403 Forbidden (Access Denied) from S3.

S3 support team says what every request should be authorized by "Authorization" header. And they advice to fix an application (s3fs in this point).

Any ideas about how to fix this?

Thanks!

Originally created by @ssvinarev on GitHub (Oct 20, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1457 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) 1.87 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.2-11 #### Kernel information (uname -r) 3.10.0-1127.19.1.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) CentOS Linux release 7.8.2003 (Core) #### s3fs command line used, if applicable ``` s3fs -d backup-cold-bucket-1:/projectname /mnt/data1 -o url=https://storage.yandexcloud.net -o endpoint=ru-central1 -o use_path_request_style -o storage_class=standard_ia -o notsup_compat_dir -o max_stat_cache_size=1000000 -o parallel_count=20 -o multipart_size=99 -o singlepart_copy_limit=-1 -o nomixupload -o use_cache=/mnt/cache -o retries=15 -o connect_timeout=60 -o readwrite_timeout=60 -o curldbg -o allow_other,uid=997,gid=99 ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` Oct 7 12:17:54 hostname s3fs[6208]: * Connection #2091 to host storage.yandexcloud.net left intact Oct 7 12:17:54 hostname s3fs[6208]: HTTP response code 200 Oct 7 12:17:57 hostname s3fs[6208]: [upload_id=0005B10F73667DBC][tpath=/stor1-data-full-odd-0301][fd=50][offset=110820259585][size=104896510] Oct 7 12:17:57 hostname s3fs[6208]: [tpath=/stor1-data-full-odd-0301][start=110820259585][size=104896510][part=1058] Oct 7 12:17:57 hostname s3fs[6208]: [tpath=/stor1-data-full-odd-0301][start=110820259585][size=104896510][part=1058] Oct 7 12:17:57 hostname s3fs[6208]: URL is https://storage.yandexcloud.net/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DB C Oct 7 12:17:57 hostname s3fs[6208]: URL changed is https://storage.yandexcloud.net/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10 F73667DBC Oct 7 12:17:58 hostname s3fs[6208]: computing signature [PUT] [/projectname/stor1-data-full-odd-0301] [partNumber=1058&uploadId=0005B10F73667DBC] [1764f1ee9310ab8495a335181c4503 61f3f6cabe1fe1ad6c8645143e5681398e] Oct 7 12:17:58 hostname s3fs[6208]: url is https://storage.yandexcloud.net Oct 7 12:17:58 hostname s3fs[6208]: * Found bundle for host storage.yandexcloud.net: 0x7fef1c0012e0 [serially] Oct 7 12:17:58 hostname s3fs[6208]: * Can not multiplex, even if we wanted to! Oct 7 12:17:58 hostname s3fs[6208]: * Re-using existing connection! (#2091) with host storage.yandexcloud.net Oct 7 12:17:58 hostname s3fs[6208]: * Connected to storage.yandexcloud.net (213.180.193.243) port 443 (#2091) Oct 7 12:17:58 hostname s3fs[6208]: > PUT /backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DBC HTTP/1.1 Oct 7 12:17:58 hostname s3fs[6208]: > Host: storage.yandexcloud.net Oct 7 12:17:58 hostname s3fs[6208]: > User-Agent: s3fs/1.87 (commit hash unknown; OpenSSL) Oct 7 12:17:58 hostname s3fs[6208]: > Authorization: AWS4-HMAC-SHA256 Credential=zxcRez4ooKDUB-KMGYQb/20201007/ru-central1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz- date, Signature=983a8bfebf7fe92af217af987840e23555a9c997b744feaac7f73955520f07e3 Oct 7 12:17:58 hostname s3fs[6208]: > x-amz-content-sha256: 1764f1ee9310ab8495a335181c450361f3f6cabe1fe1ad6c8645143e5681398e Oct 7 12:17:58 hostname s3fs[6208]: > x-amz-date: 20201007T091758Z Oct 7 12:17:58 hostname s3fs[6208]: > Content-Length: 104896510 Oct 7 12:17:58 hostname s3fs[6208]: > Expect: 100-continue Oct 7 12:17:58 hostname s3fs[6208]: > Oct 7 12:17:59 hostname s3fs[6208]: * Done waiting for 100-continue Oct 7 12:23:00 hostname s3fs[6208]: curl.cpp:CurlProgress(662): timeout now: 1602062580, curl_times[curl]: 1602062279, readwrite_timeout: 300 Oct 7 12:23:00 hostname s3fs[6208]: * Callback aborted Oct 7 12:23:00 hostname s3fs[6208]: * Closing connection 2091 Oct 7 12:23:00 hostname s3fs[6208]: curl.cpp:RequestPerform(2601): ### CURLE_ABORTED_BY_CALLBACK Oct 7 12:23:04 hostname s3fs[6208]: ### retrying... Oct 7 12:23:04 hostname s3fs[6208]: Retry request. [type=9][url=https://storage.yandexcloud.net/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DBC][path=/projectname/stor1-data-full-odd-0301] Oct 7 12:23:04 hostname s3fs[6208]: * Trying 213.180.193.243:443... Oct 7 12:23:04 hostname s3fs[6208]: * Connected to storage.yandexcloud.net (213.180.193.243) port 443 (#2092) Oct 7 12:23:04 hostname s3fs[6208]: * CAfile: none#012 CApath: none Oct 7 12:23:04 hostname s3fs[6208]: * ALPN/NPN, server did not agree to a protocol Oct 7 12:23:04 hostname s3fs[6208]: * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 Oct 7 12:23:04 hostname s3fs[6208]: * Server certificate: Oct 7 12:23:04 hostname s3fs[6208]: * subject: CN=*.website.yandexcloud.net,O=Yandex LLC,OU=ITO,L=Moscow,C=RU Oct 7 12:23:04 hostname s3fs[6208]: * start date: Oct 02 09:31:15 2020 GMT Oct 7 12:23:04 hostname s3fs[6208]: * expire date: Apr 02 09:31:15 2021 GMT Oct 7 12:23:04 hostname s3fs[6208]: * common name: *.website.yandexcloud.net Oct 7 12:23:04 hostname s3fs[6208]: * issuer: CN=Yandex CA,OU=Yandex Certification Authority,O=Yandex LLC,C=RU Oct 7 12:23:04 hostname s3fs[6208]: > PUT /backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&uploadId=0005B10F73667DBC HTTP/1.1 Oct 7 12:23:04 hostname s3fs[6208]: > Host: storage.yandexcloud.net Oct 7 12:23:04 hostname s3fs[6208]: > User-Agent: s3fs/1.87 (commit hash unknown; OpenSSL) Oct 7 12:23:04 hostname s3fs[6208]: > Accept: */* Oct 7 12:23:04 hostname s3fs[6208]: > Content-Length: 104896510 Oct 7 12:23:04 hostname s3fs[6208]: > Expect: 100-continue Oct 7 12:23:04 hostname s3fs[6208]: > Oct 7 12:23:04 hostname s3fs[6208]: * Mark bundle as not supporting multiuse Oct 7 12:23:04 hostname s3fs[6208]: < HTTP/1.1 100 Continue Oct 7 12:23:04 hostname s3fs[6208]: * Mark bundle as not supporting multiuse Oct 7 12:23:04 hostname s3fs[6208]: < HTTP/1.1 403 Forbidden Oct 7 12:23:04 hostname s3fs[6208]: < Server: nginx Oct 7 12:23:04 hostname s3fs[6208]: < Date: Wed, 07 Oct 2020 09:23:04 GMT Oct 7 12:23:04 hostname s3fs[6208]: < Content-Type: application/xml; charset=UTF-8 Oct 7 12:23:04 hostname s3fs[6208]: < Content-Length: 283 Oct 7 12:23:04 hostname s3fs[6208]: < Connection: keep-alive Oct 7 12:23:04 hostname s3fs[6208]: < Keep-Alive: timeout=60 Oct 7 12:23:04 hostname s3fs[6208]: < X-Amz-Request-Id: 55f12ff942d080f5 Oct 7 12:23:04 hostname s3fs[6208]: * HTTP error before end of send, keep sending Oct 7 12:23:04 hostname s3fs[6208]: < Oct 7 12:23:05 hostname s3fs[6208]: * We are completely uploaded and fine Oct 7 12:23:05 hostname s3fs[6208]: * Connection #2092 to host storage.yandexcloud.net left intact Oct 7 12:23:05 hostname s3fs[6208]: curl.cpp:RequestPerform(2540): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/backup-cold-bucket-1/projectname/stor1-data-full-odd-0301?partNumber=1058&amp;uploadId=0005B10F73667DBC</Resource><RequestId>55f12ff942d080f5</RequestId></Error> Oct 7 12:23:05 hostname s3fs[6208]: curl.cpp:MultipartUploadRequest(4224): failed uploading part(-1) Oct 7 12:23:05 hostname s3fs[6208]: fdcache.cpp:Write(2562): failed to multipart post(start=110820259585, size=104896510) for file(50). Oct 7 12:23:05 hostname s3fs[6208]: s3fs.cpp:s3fs_write(2240): failed to write file(/stor1-data-full-odd-0301). result=-1 ``` ### Details about issue Hello! Maybe someone can help in troubleshooting. In this log example there is attempt to upload partNumber=1058 of an object "stor1-data-full-odd-0301". This PUT-request has "Authorization" header. Fine! Request failed due to timeout. Then closing connection 2091 and retrying with a new connection 2092. Fine to! But next PUT-request doesn't have "Authorization" header and so gets response HTTP/1.1 403 Forbidden (Access Denied) from S3. S3 support team says what every request should be authorized by "Authorization" header. And they advice to fix an application (s3fs in this point). Any ideas about how to fix this? Thanks!
kerem closed this issue 2026-03-04 01:48:37 +03:00
Author
Owner

@xrefft commented on GitHub (Oct 22, 2020):

Same problem as your, It seems that there is no Authorization header for all multipart upload retry requests.

<!-- gh-comment-id:714310677 --> @xrefft commented on GitHub (Oct 22, 2020): Same problem as your, It seems that there is no `Authorization` header for all multipart upload retry requests.
Author
Owner

@ssvinarev commented on GitHub (Oct 22, 2020):

We tried to fix it by modifying curl.cpp, but no success.
Need a developer help/advice.

<!-- gh-comment-id:714317850 --> @ssvinarev commented on GitHub (Oct 22, 2020): We tried to fix it by modifying curl.cpp, but no success. Need a developer help/advice.
Author
Owner

@fly3366 commented on GitHub (Dec 22, 2020):

same as "range" header.

<!-- gh-comment-id:749312736 --> @fly3366 commented on GitHub (Dec 22, 2020): same as "range" header.
Author
Owner

@noahwilliamsson commented on GitHub (Dec 26, 2020):

Looks like this was potentially fixed with #1502 (fix: miss header when retry).

<!-- gh-comment-id:751342734 --> @noahwilliamsson commented on GitHub (Dec 26, 2020): Looks like this was potentially fixed with #1502 (fix: miss header when retry).
Author
Owner

@gaul commented on GitHub (Dec 31, 2020):

#1508 shows a way to reproduce these symptoms more reliably. It would help if a user could test with master and Chaos HTTP Proxy.

<!-- gh-comment-id:752944071 --> @gaul commented on GitHub (Dec 31, 2020): #1508 shows a way to reproduce these symptoms more reliably. It would help if a user could test with master and Chaos HTTP Proxy.
Author
Owner

@gaul commented on GitHub (Feb 8, 2021):

I believe that #1502 addresses this. Please reopen if you can reproduce the symptoms with the latest version 1.88.

<!-- gh-comment-id:775166993 --> @gaul commented on GitHub (Feb 8, 2021): I believe that #1502 addresses this. Please reopen if you can reproduce the symptoms with the latest version 1.88.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#767
No description provided.