[GH-ISSUE #1411] SignatureDoesNotMatch host s3.us-west-000.backblazeb2.com #747

Closed
opened 2026-03-04 01:48:25 +03:00 by kerem · 9 comments
Owner

Originally created by @socomsystems on GitHub (Sep 20, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1411

Been investigating this for 24 hours, no resolution yet. Have compiled on versions 1.87 and 1.85 using both openssl and gnutls, same issue resides all 4 variants.

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

example: 1.00

root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs --version
Amazon Simple Storage Service File System V1.87 (commit:unknown) with GnuTLS(gcrypt)
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

root@ark:/opt/s3fs/s3fs-fuse-1.87/src# cd ../../s3fs-fuse-1.85/src

root@ark:/opt/s3fs/s3fs-fuse-1.85/src# ./s3fs --version
Amazon Simple Storage Service File System V1.85(commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
root@ark:/opt/s3fs/s3fs-fuse-1.85/src# 

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

example: 2.9.4

root@ark:/opt/s3fs/s3fs-fuse-1.87/src#  dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 113
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 2.9.9-3
Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
 /etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
 Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
 export a virtual filesystem to the Linux kernel. It also aims to provide a
 secure method for non privileged users to create and mount their own filesystem
 implementations.
Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Homepage: https://github.com/libfuse/libfuse/wiki

Kernel information (uname -r)

command result: uname -r

5.8.0-050800-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

command result: cat /etc/os-release

NAME="Ubuntu"
VERSION="20.04.1 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.1 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

s3fs command line used, if applicable

root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs xxx /var/test2 -o passwd_file=/etc/s3fs/s3ccbb -o dbglevel=debug -f -o curldbg -o use_path_request_style -o url="https://s3.us-west-000.backblazeb2.com" > /tmp/error.out

/etc/fstab entry, if applicable

Entries as follows work fine with vultr:
s3fs#xxx /var/s3-daas/company/scbl/clients/ccy-vt fuse _netdev,allow_other,nonempty,passwd_file=/etc/s3fs/s3cc,use_path_request_style,url=https://ewr1.vultrobjects.com/ 0 0
Entries as follows fail with backblaze:
s3fs#ccb-bb /var/test2 fuse _netdev,use_path_request_style,curldbg,dbglevel=debug,allow_other,nonempty,passwd_file=/etc/s3fs/s3ccbb,url=https://s3.us-west-000.backblazeb2.com/ 0 0

There are no <CR> issues within fstab file.

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages### Details about issue

I've compiled on latest version as well as 1.85, same issue resides.

root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs xxx-bb /var/test2 -o passwd_file=/etc/s3fs/s3ccbb -o dbglevel=debug -f -o curldbg -o use_path_request_style -o url="https://s3.us-west-000.backblazeb2.com" > /tmp/error.out
root@ark:/opt/s3fs/s3fs-fuse-1.87/src# cat /tmp/error.out 
[CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [DBG] 
[INF]     s3fs.cpp:set_mountpoint_attribute(4372): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[DBG] curl.cpp:InitMimeType(695): Try to load mime types from /etc/mime.types file.
[DBG] curl.cpp:InitMimeType(700): The old mime types are cleared to load new mime types.
[INF] curl.cpp:InitMimeType(723): Loaded mime information from /etc/mime.types
[INF] fdcache.cpp:CheckCacheFileStatTopDir(134): The path to cache top dir is empty, thus not need to check permission.
[INF] s3fs.cpp:s3fs_init(3455): init v1.87(commit:unknown) with GnuTLS(gcrypt)
[INF] s3fs.cpp:s3fs_check_service(3800): check services.
[INF]       curl.cpp:CheckBucket(3527): check a bucket.
[DBG] curl.cpp:GetHandler(312): Get handler from pool: rest = 31
[INF]       curl.cpp:prepare_url(4831): URL is https://s3.us-west-000.backblazeb2.com/ccb-bb/
[INF]       curl.cpp:prepare_url(4864): URL changed is https://s3.us-west-000.backblazeb2.com/ccb-bb/
[DBG] curl.cpp:RequestPerform(2488): connecting to URL https://s3.us-west-000.backblazeb2.com/ccb-bb/
[INF]       curl.cpp:insertV4Headers(2863): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.us-west-000.backblazeb2.com
[CURL DBG] *   Trying 206.190.208.254:443...
[CURL DBG] * TCP_NODELAY set
[CURL DBG] * Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0)
[CURL DBG] * found 384 certificates in /etc/ssl/certs
[CURL DBG] * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
[CURL DBG] *      server certificate verification OK
[CURL DBG] *      server certificate status verification SKIPPED
[CURL DBG] *      common name: backblazeb2.com (matched)
[CURL DBG] *      server certificate expiration date OK
[CURL DBG] *      server certificate activation date OK
[CURL DBG] *      certificate public key: RSA
[CURL DBG] *      certificate version: #3
[CURL DBG] *      subject: CN=backblazeb2.com
[CURL DBG] *      start date: Thu, 23 Jul 2020 20:59:20 GMT
[CURL DBG] *      expire date: Wed, 21 Oct 2020 20:59:20 GMT
[CURL DBG] *      issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
[CURL DBG] > GET /xxx-bb/ HTTP/1.1
[CURL DBG] > Host: s3.us-west-000.backblazeb2.com
[CURL DBG] > User-Agent: s3fs/1.87 (commit hash unknown; GnuTLS(gcrypt))
[CURL DBG] > Accept: */*
[CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=000f78d8ac766760000000002/20200920/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ea654b4f51a95cab45f39186e8f7a34a5553f20a7663581fe3f3948dc74e3fb7
[CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[CURL DBG] > x-amz-date: 20200920T012056Z
[CURL DBG] > 
[CURL DBG] * Mark bundle as not supporting multiuse
[CURL DBG] < HTTP/1.1 403 
[CURL DBG] < Cache-Control: max-age=0, no-cache, no-store
[CURL DBG] < x-amz-request-id: f75a709861f07dae
[CURL DBG] < x-amz-id-2: adXFuv2smbstvZ3eybkY=
[CURL DBG] < Content-Type: application/xml
[CURL DBG] < Content-Length: 163
[CURL DBG] < Date: Sun, 20 Sep 2020 01:20:56 GMT
[CURL DBG] < 
[CURL DBG] * Connection #0 to host s3.us-west-000.backblazeb2.com left intact
[ERR] curl.cpp:RequestPerform(2540): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Signature validation failed</Message>
</Error>

[ERR] curl.cpp:CheckBucket(3553): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>Signature validation failed</Message>
</Error>

[CRT] s3fs.cpp:s3fs_check_service(3855): Failed to connect by sigv4, so retry to connect by signature version 2.
[DBG] curl.cpp:ReturnHandler(334): Return handler to pool
[INF] curl.cpp:ReturnHandler(341): Pool full: destroy the oldest handler
[INF]       curl.cpp:CheckBucket(3527): check a bucket.
[DBG] curl.cpp:GetHandler(312): Get handler from pool: rest = 30
[INF]       curl.cpp:prepare_url(4831): URL is https://s3.us-west-000.backblazeb2.com/ccb-bb/
[INF]       curl.cpp:prepare_url(4864): URL changed is https://s3.us-west-000.backblazeb2.com/ccb-bb/
[DBG] curl.cpp:RequestPerform(2488): connecting to URL https://s3.us-west-000.backblazeb2.com/ccb-bb/
[CURL DBG] * Found bundle for host s3.us-west-000.backblazeb2.com: 0x7fdacc001730 [serially]
[CURL DBG] * Can not multiplex, even if we wanted to!
[CURL DBG] * Re-using existing connection! (#0) with host s3.us-west-000.backblazeb2.com
[CURL DBG] * Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0)
[CURL DBG] > GET /ccb-bb/ HTTP/1.1
[CURL DBG] > Host: s3.us-west-000.backblazeb2.com
[CURL DBG] > User-Agent: s3fs/1.87 (commit hash unknown; GnuTLS(gcrypt))
[CURL DBG] > Accept: */*
[CURL DBG] > Authorization: AWS 000f78d8ac766760000000002:nKohfw/33bMoqlfrL3zLRonqRqc=
[CURL DBG] > Date: Sun, 20 Sep 2020 01:20:57 GMT
[CURL DBG] > 
[CURL DBG] * Mark bundle as not supporting multiuse
[CURL DBG] < HTTP/1.1 400 
[CURL DBG] < Cache-Control: max-age=0, no-cache, no-store
[CURL DBG] < x-amz-request-id: fcfcb15f6597c4d9
[CURL DBG] < x-amz-id-2: adRBukWvDbvNvYnddbts=
[CURL DBG] < Content-Type: application/xml
[CURL DBG] < Content-Length: 233
[CURL DBG] < Date: Sun, 20 Sep 2020 01:20:56 GMT
[CURL DBG] < Connection: close
[CURL DBG] < 
[CURL DBG] * Closing connection 0
[ERR] curl.cpp:RequestPerform(2535): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InvalidRequest</Code>
    <Message>The V2 signature authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256</Message>
</Error>

[ERR] curl.cpp:CheckBucket(3553): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InvalidRequest</Code>
    <Message>The V2 signature authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256</Message>
</Error>

[CRT] s3fs.cpp:s3fs_check_service(3867): Bad Request(host=https://s3.us-west-000.backblazeb2.com) - result of checking service.
[DBG] curl.cpp:ReturnHandler(334): Return handler to pool
[ERR] s3fs.cpp:s3fs_exit_fuseloop(3445): Exiting FUSE event loop due to errors

[INF] s3fs.cpp:s3fs_destroy(3513): destroy
[WAN] s3fs.cpp:s3fs_destroy(3517): Failed to clean up signal object.

Originally created by @socomsystems on GitHub (Sep 20, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1411 Been investigating this for 24 hours, no resolution yet. Have compiled on versions 1.87 and 1.85 using both openssl and gnutls, same issue resides all 4 variants. ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) _example: 1.00_ ``` root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs --version Amazon Simple Storage Service File System V1.87 (commit:unknown) with GnuTLS(gcrypt) Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. root@ark:/opt/s3fs/s3fs-fuse-1.87/src# cd ../../s3fs-fuse-1.85/src root@ark:/opt/s3fs/s3fs-fuse-1.85/src# ./s3fs --version Amazon Simple Storage Service File System V1.85(commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. root@ark:/opt/s3fs/s3fs-fuse-1.85/src# ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _example: 2.9.4_ ``` root@ark:/opt/s3fs/s3fs-fuse-1.87/src# dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 113 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Architecture: amd64 Version: 2.9.9-3 Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Homepage: https://github.com/libfuse/libfuse/wiki ``` #### Kernel information (uname -r) _command result: uname -r_ ``` 5.8.0-050800-generic ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) _command result: cat /etc/os-release_ ``` NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.1 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal ``` #### s3fs command line used, if applicable ``` root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs xxx /var/test2 -o passwd_file=/etc/s3fs/s3ccbb -o dbglevel=debug -f -o curldbg -o use_path_request_style -o url="https://s3.us-west-000.backblazeb2.com" > /tmp/error.out ``` #### /etc/fstab entry, if applicable ``` Entries as follows work fine with vultr: s3fs#xxx /var/s3-daas/company/scbl/clients/ccy-vt fuse _netdev,allow_other,nonempty,passwd_file=/etc/s3fs/s3cc,use_path_request_style,url=https://ewr1.vultrobjects.com/ 0 0 Entries as follows fail with backblaze: s3fs#ccb-bb /var/test2 fuse _netdev,use_path_request_style,curldbg,dbglevel=debug,allow_other,nonempty,passwd_file=/etc/s3fs/s3ccbb,url=https://s3.us-west-000.backblazeb2.com/ 0 0 There are no <CR> issues within fstab file. ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_### Details about issue ``` I've compiled on latest version as well as 1.85, same issue resides. root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs xxx-bb /var/test2 -o passwd_file=/etc/s3fs/s3ccbb -o dbglevel=debug -f -o curldbg -o use_path_request_style -o url="https://s3.us-west-000.backblazeb2.com" > /tmp/error.out root@ark:/opt/s3fs/s3fs-fuse-1.87/src# cat /tmp/error.out [CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [DBG] [INF] s3fs.cpp:set_mountpoint_attribute(4372): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [DBG] curl.cpp:InitMimeType(695): Try to load mime types from /etc/mime.types file. [DBG] curl.cpp:InitMimeType(700): The old mime types are cleared to load new mime types. [INF] curl.cpp:InitMimeType(723): Loaded mime information from /etc/mime.types [INF] fdcache.cpp:CheckCacheFileStatTopDir(134): The path to cache top dir is empty, thus not need to check permission. [INF] s3fs.cpp:s3fs_init(3455): init v1.87(commit:unknown) with GnuTLS(gcrypt) [INF] s3fs.cpp:s3fs_check_service(3800): check services. [INF] curl.cpp:CheckBucket(3527): check a bucket. [DBG] curl.cpp:GetHandler(312): Get handler from pool: rest = 31 [INF] curl.cpp:prepare_url(4831): URL is https://s3.us-west-000.backblazeb2.com/ccb-bb/ [INF] curl.cpp:prepare_url(4864): URL changed is https://s3.us-west-000.backblazeb2.com/ccb-bb/ [DBG] curl.cpp:RequestPerform(2488): connecting to URL https://s3.us-west-000.backblazeb2.com/ccb-bb/ [INF] curl.cpp:insertV4Headers(2863): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.us-west-000.backblazeb2.com [CURL DBG] * Trying 206.190.208.254:443... [CURL DBG] * TCP_NODELAY set [CURL DBG] * Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0) [CURL DBG] * found 384 certificates in /etc/ssl/certs [CURL DBG] * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 [CURL DBG] * server certificate verification OK [CURL DBG] * server certificate status verification SKIPPED [CURL DBG] * common name: backblazeb2.com (matched) [CURL DBG] * server certificate expiration date OK [CURL DBG] * server certificate activation date OK [CURL DBG] * certificate public key: RSA [CURL DBG] * certificate version: #3 [CURL DBG] * subject: CN=backblazeb2.com [CURL DBG] * start date: Thu, 23 Jul 2020 20:59:20 GMT [CURL DBG] * expire date: Wed, 21 Oct 2020 20:59:20 GMT [CURL DBG] * issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3 [CURL DBG] > GET /xxx-bb/ HTTP/1.1 [CURL DBG] > Host: s3.us-west-000.backblazeb2.com [CURL DBG] > User-Agent: s3fs/1.87 (commit hash unknown; GnuTLS(gcrypt)) [CURL DBG] > Accept: */* [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=000f78d8ac766760000000002/20200920/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=ea654b4f51a95cab45f39186e8f7a34a5553f20a7663581fe3f3948dc74e3fb7 [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 [CURL DBG] > x-amz-date: 20200920T012056Z [CURL DBG] > [CURL DBG] * Mark bundle as not supporting multiuse [CURL DBG] < HTTP/1.1 403 [CURL DBG] < Cache-Control: max-age=0, no-cache, no-store [CURL DBG] < x-amz-request-id: f75a709861f07dae [CURL DBG] < x-amz-id-2: adXFuv2smbstvZ3eybkY= [CURL DBG] < Content-Type: application/xml [CURL DBG] < Content-Length: 163 [CURL DBG] < Date: Sun, 20 Sep 2020 01:20:56 GMT [CURL DBG] < [CURL DBG] * Connection #0 to host s3.us-west-000.backblazeb2.com left intact [ERR] curl.cpp:RequestPerform(2540): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Error> <Code>SignatureDoesNotMatch</Code> <Message>Signature validation failed</Message> </Error> [ERR] curl.cpp:CheckBucket(3553): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Error> <Code>SignatureDoesNotMatch</Code> <Message>Signature validation failed</Message> </Error> [CRT] s3fs.cpp:s3fs_check_service(3855): Failed to connect by sigv4, so retry to connect by signature version 2. [DBG] curl.cpp:ReturnHandler(334): Return handler to pool [INF] curl.cpp:ReturnHandler(341): Pool full: destroy the oldest handler [INF] curl.cpp:CheckBucket(3527): check a bucket. [DBG] curl.cpp:GetHandler(312): Get handler from pool: rest = 30 [INF] curl.cpp:prepare_url(4831): URL is https://s3.us-west-000.backblazeb2.com/ccb-bb/ [INF] curl.cpp:prepare_url(4864): URL changed is https://s3.us-west-000.backblazeb2.com/ccb-bb/ [DBG] curl.cpp:RequestPerform(2488): connecting to URL https://s3.us-west-000.backblazeb2.com/ccb-bb/ [CURL DBG] * Found bundle for host s3.us-west-000.backblazeb2.com: 0x7fdacc001730 [serially] [CURL DBG] * Can not multiplex, even if we wanted to! [CURL DBG] * Re-using existing connection! (#0) with host s3.us-west-000.backblazeb2.com [CURL DBG] * Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0) [CURL DBG] > GET /ccb-bb/ HTTP/1.1 [CURL DBG] > Host: s3.us-west-000.backblazeb2.com [CURL DBG] > User-Agent: s3fs/1.87 (commit hash unknown; GnuTLS(gcrypt)) [CURL DBG] > Accept: */* [CURL DBG] > Authorization: AWS 000f78d8ac766760000000002:nKohfw/33bMoqlfrL3zLRonqRqc= [CURL DBG] > Date: Sun, 20 Sep 2020 01:20:57 GMT [CURL DBG] > [CURL DBG] * Mark bundle as not supporting multiuse [CURL DBG] < HTTP/1.1 400 [CURL DBG] < Cache-Control: max-age=0, no-cache, no-store [CURL DBG] < x-amz-request-id: fcfcb15f6597c4d9 [CURL DBG] < x-amz-id-2: adRBukWvDbvNvYnddbts= [CURL DBG] < Content-Type: application/xml [CURL DBG] < Content-Length: 233 [CURL DBG] < Date: Sun, 20 Sep 2020 01:20:56 GMT [CURL DBG] < Connection: close [CURL DBG] < [CURL DBG] * Closing connection 0 [ERR] curl.cpp:RequestPerform(2535): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Error> <Code>InvalidRequest</Code> <Message>The V2 signature authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256</Message> </Error> [ERR] curl.cpp:CheckBucket(3553): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <Error> <Code>InvalidRequest</Code> <Message>The V2 signature authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256</Message> </Error> [CRT] s3fs.cpp:s3fs_check_service(3867): Bad Request(host=https://s3.us-west-000.backblazeb2.com) - result of checking service. [DBG] curl.cpp:ReturnHandler(334): Return handler to pool [ERR] s3fs.cpp:s3fs_exit_fuseloop(3445): Exiting FUSE event loop due to errors [INF] s3fs.cpp:s3fs_destroy(3513): destroy [WAN] s3fs.cpp:s3fs_destroy(3517): Failed to clean up signal object. ```
kerem 2026-03-04 01:48:25 +03:00
  • closed this issue
  • added the
    need info
    label
Author
Owner

@gaul commented on GitHub (Sep 20, 2020):

Can you test your credentials using the AWS CLI? First you need to add your B2 S3 credentials (these are different than your B2 credentials) to $HOME/.aws/credentials:

aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxx
aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Next list your buckets:

$ aws --profile=backblaze --endpoint=https://s3.us-west-001.backblazeb2.com s3 ls
2020-05-12 22:43:14 gauls3

I successfully mounted the bucket afterwards with s3fs.

<!-- gh-comment-id:695767310 --> @gaul commented on GitHub (Sep 20, 2020): Can you test your credentials using the AWS CLI? First you need to add your B2 S3 credentials (these are different than your B2 credentials) to `$HOME/.aws/credentials`: ``` aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxxxxx aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` Next list your buckets: ``` $ aws --profile=backblaze --endpoint=https://s3.us-west-001.backblazeb2.com s3 ls 2020-05-12 22:43:14 gauls3 ``` I successfully mounted the bucket afterwards with s3fs.
Author
Owner

@AlexTalker commented on GitHub (Sep 20, 2020):

@gaul I am too have struggle with connecting to B2 via s3fs-fuse, but now
I filled in the command though(you missed [backblaze] in config file example thou)
and it works and then I filled in credentials in the mount example here and it works too.
I don't exactly get what I did wrong past time,
so this all gets interesting.
I think on some point I have seen the SignatureDoesNotMatch error too but I quickly went trying out other solutions.
I used EPEL package by the way.

<!-- gh-comment-id:695795081 --> @AlexTalker commented on GitHub (Sep 20, 2020): @gaul I am too have struggle with connecting to B2 via `s3fs-fuse`, but now I filled in the command though(you missed `[backblaze]` in config file example thou) and it works and then I filled in credentials in the mount example here and it works too. I don't exactly get what I did wrong past time, so this all gets interesting. I think on some point I have seen the SignatureDoesNotMatch error too but I quickly went trying out other solutions. I used EPEL package by the way.
Author
Owner

@AlexTalker commented on GitHub (Sep 20, 2020):

This all is so confusing, especially when following the official guide ( https://help.backblaze.com/hc/en-us/articles/360047773653-Using-S3FS-with-B2 ) and it DIDN'T work probably because they skipped -o use_path_request_style

<!-- gh-comment-id:695795249 --> @AlexTalker commented on GitHub (Sep 20, 2020): This all is so confusing, especially when following the official guide ( https://help.backblaze.com/hc/en-us/articles/360047773653-Using-S3FS-with-B2 ) and it DIDN'T work probably because they skipped `-o use_path_request_style`
Author
Owner

@socomsystems commented on GitHub (Sep 20, 2020):

@gaul I gave aws cli a stabb and it worked with no issue.

I then tried several different permutation orders of command line s3fs as installed v1.85 - all failed

I then went back to v1.87, recompiled but this time with both --with-gnutls and --with-nettle ...and snap, its looks like we're getting somewhere. Passed the cipher match, no multiuse support (bb may not support mus if I recall). Really odd though, something to do with libcurl and/or openssl is a misfire.

I'll keep at it. Anyone else able to reproduce what I'm seeing?

root@ark:/opt/s3fs/s3fs-fuse-1.87/src# nslookup s3.us-west-000.backblazeb2.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	s3.us-west-000.backblazeb2.com
Address: 206.190.208.254

root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs ccb-bb /var/test2 -o passwd_file=/etc/s3fs/s3ccbb -o dbglevel=debug -f -o curldbg -o use_path_request_style -o url=https://s3.us-west-000.backblazeb2.com/
[CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [DBG] 
[INF]     s3fs.cpp:set_mountpoint_attribute(4372): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[DBG] curl.cpp:InitMimeType(695): Try to load mime types from /etc/mime.types file.
[DBG] curl.cpp:InitMimeType(700): The old mime types are cleared to load new mime types.
[INF] curl.cpp:InitMimeType(723): Loaded mime information from /etc/mime.types
[INF] fdcache.cpp:CheckCacheFileStatTopDir(134): The path to cache top dir is empty, thus not need to check permission.
[INF] s3fs.cpp:s3fs_init(3455): init v1.87(commit:unknown) with GnuTLS(nettle)
[INF] s3fs.cpp:s3fs_check_service(3800): check services.
[INF]       curl.cpp:CheckBucket(3527): check a bucket.
[DBG] curl.cpp:GetHandler(312): Get handler from pool: rest = 31
[INF]       curl.cpp:prepare_url(4831): URL is https://s3.us-west-000.backblazeb2.com/ccb-bb/
[INF]       curl.cpp:prepare_url(4864): URL changed is https://s3.us-west-000.backblazeb2.com/ccb-bb/
[DBG] curl.cpp:RequestPerform(2488): connecting to URL https://s3.us-west-000.backblazeb2.com/ccb-bb/
[INF]       curl.cpp:insertV4Headers(2863): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(99): url is https://s3.us-west-000.backblazeb2.com
[CURL DBG] *   Trying 206.190.208.254:443...
[CURL DBG] * TCP_NODELAY set
[CURL DBG] * Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0)
[CURL DBG] * found 384 certificates in /etc/ssl/certs
[CURL DBG] * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
[CURL DBG] *      server certificate verification OK
[CURL DBG] *      server certificate status verification SKIPPED
[CURL DBG] *      common name: backblazeb2.com (matched)
[CURL DBG] *      server certificate expiration date OK
[CURL DBG] *      server certificate activation date OK
[CURL DBG] *      certificate public key: RSA
[CURL DBG] *      certificate version: #3
[CURL DBG] *      subject: CN=backblazeb2.com
[CURL DBG] *      start date: Thu, 23 Jul 2020 20:59:20 GMT
[CURL DBG] *      expire date: Wed, 21 Oct 2020 20:59:20 GMT
[CURL DBG] *      issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
[CURL DBG] > GET /ccb-bb/ HTTP/1.1
[CURL DBG] > Host: s3.us-west-000.backblazeb2.com
[CURL DBG] > User-Agent: s3fs/1.87 (commit hash unknown; GnuTLS(nettle))
[CURL DBG] > Accept: */*
[CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=000f78d8ac766760000000008/20200920/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fd26f30efdea0dd98ff29313fc08a90fdada9562aaa0938a020b044cdfb42e83
[CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[CURL DBG] > x-amz-date: 20200920T210900Z
[CURL DBG] > 
[CURL DBG] * Mark bundle as not supporting multiuse
[CURL DBG] < HTTP/1.1 200 
[CURL DBG] < Cache-Control: max-age=0, no-cache, no-store
[CURL DBG] < x-amz-request-id: ce9b22b25cbf84ab
[CURL DBG] < x-amz-id-2: aZqU30zjXZMU48GF/Y1E3njaCNpI3ejZm
[CURL DBG] < Content-Type: application/xml
[CURL DBG] < Content-Length: 425479
[CURL DBG] < Date: Sun, 20 Sep 2020 21:09:00 GMT
[CURL DBG] < 
[CURL DBG] * Connection #0 to host s3.us-west-000.backblazeb2.com left intact
[INF]       curl.cpp:RequestPerform(2520): HTTP response code 200
[DBG] curl.cpp:ReturnHandler(334): Return handler to pool
[INF] curl.cpp:ReturnHandler(341): Pool full: destroy the oldest handler
[ERR] s3fs.cpp:s3fs_init(3505): Failed to initialize signal object, but continue...
<!-- gh-comment-id:695838271 --> @socomsystems commented on GitHub (Sep 20, 2020): @gaul I gave aws cli a stabb and it worked with no issue. I then tried several different permutation orders of command line s3fs as installed v1.85 - all failed I then went back to v1.87, recompiled but this time with both --with-gnutls and --with-nettle ...and snap, its looks like we're getting somewhere. Passed the cipher match, no multiuse support (bb may not support mus if I recall). Really odd though, something to do with libcurl and/or openssl is a misfire. I'll keep at it. Anyone else able to reproduce what I'm seeing? ``` root@ark:/opt/s3fs/s3fs-fuse-1.87/src# nslookup s3.us-west-000.backblazeb2.com Server: 127.0.0.53 Address: 127.0.0.53#53 Non-authoritative answer: Name: s3.us-west-000.backblazeb2.com Address: 206.190.208.254 root@ark:/opt/s3fs/s3fs-fuse-1.87/src# ./s3fs ccb-bb /var/test2 -o passwd_file=/etc/s3fs/s3ccbb -o dbglevel=debug -f -o curldbg -o use_path_request_style -o url=https://s3.us-west-000.backblazeb2.com/ [CRT] sighandlers.cpp:SetLogLevel(168): change debug level from [CRT] to [DBG] [INF] s3fs.cpp:set_mountpoint_attribute(4372): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [DBG] curl.cpp:InitMimeType(695): Try to load mime types from /etc/mime.types file. [DBG] curl.cpp:InitMimeType(700): The old mime types are cleared to load new mime types. [INF] curl.cpp:InitMimeType(723): Loaded mime information from /etc/mime.types [INF] fdcache.cpp:CheckCacheFileStatTopDir(134): The path to cache top dir is empty, thus not need to check permission. [INF] s3fs.cpp:s3fs_init(3455): init v1.87(commit:unknown) with GnuTLS(nettle) [INF] s3fs.cpp:s3fs_check_service(3800): check services. [INF] curl.cpp:CheckBucket(3527): check a bucket. [DBG] curl.cpp:GetHandler(312): Get handler from pool: rest = 31 [INF] curl.cpp:prepare_url(4831): URL is https://s3.us-west-000.backblazeb2.com/ccb-bb/ [INF] curl.cpp:prepare_url(4864): URL changed is https://s3.us-west-000.backblazeb2.com/ccb-bb/ [DBG] curl.cpp:RequestPerform(2488): connecting to URL https://s3.us-west-000.backblazeb2.com/ccb-bb/ [INF] curl.cpp:insertV4Headers(2863): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(99): url is https://s3.us-west-000.backblazeb2.com [CURL DBG] * Trying 206.190.208.254:443... [CURL DBG] * TCP_NODELAY set [CURL DBG] * Connected to s3.us-west-000.backblazeb2.com (206.190.208.254) port 443 (#0) [CURL DBG] * found 384 certificates in /etc/ssl/certs [CURL DBG] * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 [CURL DBG] * server certificate verification OK [CURL DBG] * server certificate status verification SKIPPED [CURL DBG] * common name: backblazeb2.com (matched) [CURL DBG] * server certificate expiration date OK [CURL DBG] * server certificate activation date OK [CURL DBG] * certificate public key: RSA [CURL DBG] * certificate version: #3 [CURL DBG] * subject: CN=backblazeb2.com [CURL DBG] * start date: Thu, 23 Jul 2020 20:59:20 GMT [CURL DBG] * expire date: Wed, 21 Oct 2020 20:59:20 GMT [CURL DBG] * issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3 [CURL DBG] > GET /ccb-bb/ HTTP/1.1 [CURL DBG] > Host: s3.us-west-000.backblazeb2.com [CURL DBG] > User-Agent: s3fs/1.87 (commit hash unknown; GnuTLS(nettle)) [CURL DBG] > Accept: */* [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=000f78d8ac766760000000008/20200920/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=fd26f30efdea0dd98ff29313fc08a90fdada9562aaa0938a020b044cdfb42e83 [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 [CURL DBG] > x-amz-date: 20200920T210900Z [CURL DBG] > [CURL DBG] * Mark bundle as not supporting multiuse [CURL DBG] < HTTP/1.1 200 [CURL DBG] < Cache-Control: max-age=0, no-cache, no-store [CURL DBG] < x-amz-request-id: ce9b22b25cbf84ab [CURL DBG] < x-amz-id-2: aZqU30zjXZMU48GF/Y1E3njaCNpI3ejZm [CURL DBG] < Content-Type: application/xml [CURL DBG] < Content-Length: 425479 [CURL DBG] < Date: Sun, 20 Sep 2020 21:09:00 GMT [CURL DBG] < [CURL DBG] * Connection #0 to host s3.us-west-000.backblazeb2.com left intact [INF] curl.cpp:RequestPerform(2520): HTTP response code 200 [DBG] curl.cpp:ReturnHandler(334): Return handler to pool [INF] curl.cpp:ReturnHandler(341): Pool full: destroy the oldest handler [ERR] s3fs.cpp:s3fs_init(3505): Failed to initialize signal object, but continue... ```
Author
Owner

@socomsystems commented on GitHub (Sep 20, 2020):

Ok, openssl issue confirmed. I reconfigured as such only: ./configure --prefix=/usr --with-gnutls ...no other flags on 1.87 and was able to get as far as the last posts log. Failed to mention one package may have been missing as I did execute the following apt commands. If so, I didn't see it or was distracted.

344 apt-get install build-essential libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support automake libtool
347 apt install libcurl4-gnutls-dev
350 apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config
<!-- gh-comment-id:695839669 --> @socomsystems commented on GitHub (Sep 20, 2020): Ok, openssl issue confirmed. I reconfigured as such only: ./configure --prefix=/usr --with-gnutls ...no other flags on 1.87 and was able to get as far as the last posts log. Failed to mention one package may have been missing as I did execute the following apt commands. If so, I didn't see it or was distracted. ``` 344 apt-get install build-essential libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support automake libtool 347 apt install libcurl4-gnutls-dev 350 apt-get install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config ```
Author
Owner

@gaul commented on GitHub (Sep 21, 2020):

@socomsystems Great debugging! S3 servers emit SignatureDoesNotMatch when either the headers or signature of the headers do not match. I wonder if OpenSSL is adding or overwriting a header? FWIW I successfully ran s3fs using OpenSSL 1.1.1g on Fedora 32.

<!-- gh-comment-id:695874723 --> @gaul commented on GitHub (Sep 21, 2020): @socomsystems Great debugging! S3 servers emit `SignatureDoesNotMatch` when either the headers or signature of the headers do not match. I wonder if OpenSSL is adding or overwriting a header? FWIW I successfully ran s3fs using OpenSSL 1.1.1g on Fedora 32.
Author
Owner

@socomsystems commented on GitHub (Sep 21, 2020):

Final Backblaze s3fs v1.87 in optimized for my environment fstab form factor

s3fs#cczbb /var/s3-daas/company/scbl/clients/cczbb fuse _netdev,cipher_suites=AESGCM,max_background=1000,max_stat_cache_size=100000,multipart_size=52,parallel_count=30,multireq_max=30,dbglevel=warn,use_cache=/BucketsOfCash,retries=7,allow_other,nonempty,passwd_file=/etc/s3fs/ccbb,use_path_request_style,url=https://s3.us-west-000.backblazeb2.com/ 0 0

A possible perp to include in the lineup looks to be OpenSSL 1.1.1f 31 Mar 2020, libcurl and/or keys crossover. Not looking any further problem solved, closing thread. Thanks for chiming in guys.

<!-- gh-comment-id:695883180 --> @socomsystems commented on GitHub (Sep 21, 2020): **Final Backblaze s3fs v1.87 in optimized for my environment fstab form factor** ``` s3fs#cczbb /var/s3-daas/company/scbl/clients/cczbb fuse _netdev,cipher_suites=AESGCM,max_background=1000,max_stat_cache_size=100000,multipart_size=52,parallel_count=30,multireq_max=30,dbglevel=warn,use_cache=/BucketsOfCash,retries=7,allow_other,nonempty,passwd_file=/etc/s3fs/ccbb,use_path_request_style,url=https://s3.us-west-000.backblazeb2.com/ 0 0 ``` A possible perp to include in the lineup looks to be OpenSSL 1.1.1f 31 Mar 2020, libcurl and/or keys crossover. Not looking any further problem solved, closing thread. Thanks for chiming in guys.
Author
Owner

@arichiardi commented on GitHub (May 4, 2021):

@socomsystems thanks for debugging this!

Was the problem with OpenSSL 1.1.1f? I having the same problem within Clonezilla Live.

<!-- gh-comment-id:832293908 --> @arichiardi commented on GitHub (May 4, 2021): @socomsystems thanks for debugging this! Was the problem with `OpenSSL 1.1.1f`? I having the same problem within Clonezilla Live.
Author
Owner

@Telofy commented on GitHub (Jul 23, 2021):

FYI: I’ve had similar issues (though my error was InvalidAccessKeyId) and tried to compile s3fs with OpenSSL, Nettle, and GNUTLS to no avail. The solution was to create a new key instead of the “Master Application Key” and use the new one.

<!-- gh-comment-id:885732284 --> @Telofy commented on GitHub (Jul 23, 2021): FYI: I’ve had similar issues (though my error was `InvalidAccessKeyId`) and tried to compile s3fs with OpenSSL, Nettle, and GNUTLS to no avail. The solution was to create a new key instead of the “Master Application Key” and use the new one.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#747
No description provided.