[GH-ISSUE #1626] Cannot mount S3 compatible storage (NetApp) #853

Closed
opened 2026-03-04 01:49:22 +03:00 by kerem · 7 comments
Owner

Originally created by @nicoske on GitHub (Apr 20, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1626

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.89 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Name : fuse
Version : 2.9.7
Release : 12.el8
Architecture: x86_64

Kernel information (uname -r)

4.18.0-240.15.1.el8_3.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Red Hat Enterprise Linux"
VERSION="8.3 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.3"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.3"

s3fs command line used, if applicable

s3fs bucket-1 /mnt/s3 -o passwd_file=/etc/passwd-s3fs -o url=https://notdisclosed -o use_path_request_style,no_check_certificate -o f2 -d -d -f

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

2021-04-20T09:23:43.623Z [CRT] s3fs_logger.cpp:LowSetLogLevel(219): change debug level from [CRT] to [DBG]
2021-04-20T09:23:43.623Z [INF]     s3fs.cpp:set_mountpoint_attribute(4020): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777)
2021-04-20T09:23:43.624Z [DBG] curl.cpp:InitMimeType(408): Try to load mime types from /etc/mime.types file.
2021-04-20T09:23:43.624Z [DBG] curl.cpp:InitMimeType(413): The old mime types are cleared to load new mime types.
2021-04-20T09:23:43.628Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types
2021-04-20T09:23:43.628Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission.
FUSE library version: 2.9.7
nullpath_ok: 0
nopath: 0
utime_omit_ok: 0
unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0
INIT: 7.31
flags=0x03fffffb
max_readahead=0x00020000
2021-04-20T09:23:43.635Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with OpenSSL
2021-04-20T09:23:43.635Z [INF] s3fs.cpp:s3fs_check_service(3447): check services.
2021-04-20T09:23:43.635Z [INF]       curl.cpp:CheckBucket(3364): check a bucket.
2021-04-20T09:23:43.635Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-04-20T09:23:43.635Z [DBG] curl.cpp:ResetHandle(1935): 'no_check_certificate' option in effect.
2021-04-20T09:23:43.635Z [DBG] curl.cpp:ResetHandle(1936): The server certificate won't be checked against the available certificate authorities.
2021-04-20T09:23:43.635Z [INF]       curl_util.cpp:prepare_url(250): URL is https://notdisclosed/bucket-1/
2021-04-20T09:23:43.635Z [INF]       curl_util.cpp:prepare_url(283): URL changed is https://notdisclosed/bucket-1/
2021-04-20T09:23:43.635Z [DBG] curl.cpp:RequestPerform(2254): connecting to URL https://notdisclosed/bucket-1/
2021-04-20T09:23:43.635Z [INF]       curl.cpp:insertV4Headers(2640): computing signature [GET] [/] [] []
2021-04-20T09:23:43.635Z [INF]       curl_util.cpp:url_to_host(327): url is https://notdisclosed
2021-04-20T09:23:43.671Z [ERR] curl.cpp:RequestPerform(2324): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message></Error>
2021-04-20T09:23:43.671Z [ERR] curl.cpp:CheckBucket(3396): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message></Error>
2021-04-20T09:23:43.671Z [CRT] s3fs.cpp:s3fs_check_service(3502): Failed to connect by sigv4, so retry to connect by signature version 2.
2021-04-20T09:23:43.671Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2021-04-20T09:23:43.671Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2021-04-20T09:23:43.671Z [INF]       curl.cpp:CheckBucket(3364): check a bucket.
2021-04-20T09:23:43.671Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30
2021-04-20T09:23:43.671Z [DBG] curl.cpp:ResetHandle(1935): 'no_check_certificate' option in effect.
2021-04-20T09:23:43.671Z [DBG] curl.cpp:ResetHandle(1936): The server certificate won't be checked against the available certificate authorities.
2021-04-20T09:23:43.671Z [INF]       curl_util.cpp:prepare_url(250): URL is https://notdisclosed/bucket-1/
2021-04-20T09:23:43.671Z [INF]       curl_util.cpp:prepare_url(283): URL changed is https://notdisclosed/bucket-1/
2021-04-20T09:23:43.671Z [DBG] curl.cpp:RequestPerform(2254): connecting to URL https://notdisclosed/bucket-1/
2021-04-20T09:23:43.699Z [ERR] curl.cpp:RequestPerform(2318): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message></Error>
2021-04-20T09:23:43.699Z [ERR] curl.cpp:CheckBucket(3396): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message></Error>
2021-04-20T09:23:43.699Z [CRT] s3fs.cpp:s3fs_check_service(3514): Bad Request(host=https://notdisclosed) - result of checking service.
2021-04-20T09:23:43.699Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2021-04-20T09:23:43.699Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3321): Exiting FUSE event loop due to errors

   INIT: 7.19
   flags=0x00000011
   max_readahead=0x00020000
   max_write=0x00020000
   max_background=0
   congestion_threshold=0
   unique: 2, success, outsize: 40
2021-04-20T09:23:43.704Z [INF] s3fs.cpp:s3fs_destroy(3389): destroy

Details about issue

I cannot mount S3 compatible storage (NetApp) using s3fs. Their implementation require to enable path style and only support signature v4.

It works fine with s3cmd, Wowza, S3 browser.

Originally created by @nicoske on GitHub (Apr 20, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1626 ### Additional Information #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.89 (commit:unknown) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Name : fuse Version : 2.9.7 Release : 12.el8 Architecture: x86_64 #### Kernel information (uname -r) 4.18.0-240.15.1.el8_3.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Red Hat Enterprise Linux" VERSION="8.3 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.3" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.3 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8.3:GA" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.3" #### s3fs command line used, if applicable ``` s3fs bucket-1 /mnt/s3 -o passwd_file=/etc/passwd-s3fs -o url=https://notdisclosed -o use_path_request_style,no_check_certificate -o f2 -d -d -f ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` 2021-04-20T09:23:43.623Z [CRT] s3fs_logger.cpp:LowSetLogLevel(219): change debug level from [CRT] to [DBG] 2021-04-20T09:23:43.623Z [INF] s3fs.cpp:set_mountpoint_attribute(4020): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777) 2021-04-20T09:23:43.624Z [DBG] curl.cpp:InitMimeType(408): Try to load mime types from /etc/mime.types file. 2021-04-20T09:23:43.624Z [DBG] curl.cpp:InitMimeType(413): The old mime types are cleared to load new mime types. 2021-04-20T09:23:43.628Z [INF] curl.cpp:InitMimeType(436): Loaded mime information from /etc/mime.types 2021-04-20T09:23:43.628Z [INF] fdcache_stat.cpp:CheckCacheFileStatTopDir(79): The path to cache top dir is empty, thus not need to check permission. FUSE library version: 2.9.7 nullpath_ok: 0 nopath: 0 utime_omit_ok: 0 unique: 2, opcode: INIT (26), nodeid: 0, insize: 56, pid: 0 INIT: 7.31 flags=0x03fffffb max_readahead=0x00020000 2021-04-20T09:23:43.635Z [INF] s3fs.cpp:s3fs_init(3331): init v1.89(commit:unknown) with OpenSSL 2021-04-20T09:23:43.635Z [INF] s3fs.cpp:s3fs_check_service(3447): check services. 2021-04-20T09:23:43.635Z [INF] curl.cpp:CheckBucket(3364): check a bucket. 2021-04-20T09:23:43.635Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31 2021-04-20T09:23:43.635Z [DBG] curl.cpp:ResetHandle(1935): 'no_check_certificate' option in effect. 2021-04-20T09:23:43.635Z [DBG] curl.cpp:ResetHandle(1936): The server certificate won't be checked against the available certificate authorities. 2021-04-20T09:23:43.635Z [INF] curl_util.cpp:prepare_url(250): URL is https://notdisclosed/bucket-1/ 2021-04-20T09:23:43.635Z [INF] curl_util.cpp:prepare_url(283): URL changed is https://notdisclosed/bucket-1/ 2021-04-20T09:23:43.635Z [DBG] curl.cpp:RequestPerform(2254): connecting to URL https://notdisclosed/bucket-1/ 2021-04-20T09:23:43.635Z [INF] curl.cpp:insertV4Headers(2640): computing signature [GET] [/] [] [] 2021-04-20T09:23:43.635Z [INF] curl_util.cpp:url_to_host(327): url is https://notdisclosed 2021-04-20T09:23:43.671Z [ERR] curl.cpp:RequestPerform(2324): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message></Error> 2021-04-20T09:23:43.671Z [ERR] curl.cpp:CheckBucket(3396): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message></Error> 2021-04-20T09:23:43.671Z [CRT] s3fs.cpp:s3fs_check_service(3502): Failed to connect by sigv4, so retry to connect by signature version 2. 2021-04-20T09:23:43.671Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool 2021-04-20T09:23:43.671Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler 2021-04-20T09:23:43.671Z [INF] curl.cpp:CheckBucket(3364): check a bucket. 2021-04-20T09:23:43.671Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 30 2021-04-20T09:23:43.671Z [DBG] curl.cpp:ResetHandle(1935): 'no_check_certificate' option in effect. 2021-04-20T09:23:43.671Z [DBG] curl.cpp:ResetHandle(1936): The server certificate won't be checked against the available certificate authorities. 2021-04-20T09:23:43.671Z [INF] curl_util.cpp:prepare_url(250): URL is https://notdisclosed/bucket-1/ 2021-04-20T09:23:43.671Z [INF] curl_util.cpp:prepare_url(283): URL changed is https://notdisclosed/bucket-1/ 2021-04-20T09:23:43.671Z [DBG] curl.cpp:RequestPerform(2254): connecting to URL https://notdisclosed/bucket-1/ 2021-04-20T09:23:43.699Z [ERR] curl.cpp:RequestPerform(2318): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message></Error> 2021-04-20T09:23:43.699Z [ERR] curl.cpp:CheckBucket(3396): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message></Error> 2021-04-20T09:23:43.699Z [CRT] s3fs.cpp:s3fs_check_service(3514): Bad Request(host=https://notdisclosed) - result of checking service. 2021-04-20T09:23:43.699Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool 2021-04-20T09:23:43.699Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3321): Exiting FUSE event loop due to errors INIT: 7.19 flags=0x00000011 max_readahead=0x00020000 max_write=0x00020000 max_background=0 congestion_threshold=0 unique: 2, success, outsize: 40 2021-04-20T09:23:43.704Z [INF] s3fs.cpp:s3fs_destroy(3389): destroy ``` ### Details about issue I cannot mount S3 compatible storage (NetApp) using s3fs. Their implementation require to enable path style and only support signature v4. It works fine with s3cmd, Wowza, S3 browser.
kerem closed this issue 2026-03-04 01:49:23 +03:00
Author
Owner

@gaul commented on GitHub (Apr 20, 2021):

It appears that your credentials are incorrect due to the SignatureDoesNotMatch error code. Can you see if they are what you expect in $HOME/.aws/credentials? You might try testing with the AWS CLI since s3fs supports the same format.

<!-- gh-comment-id:823293080 --> @gaul commented on GitHub (Apr 20, 2021): It appears that your credentials are incorrect due to the `SignatureDoesNotMatch` error code. Can you see if they are what you expect in `$HOME/.aws/credentials`? You might try testing with the AWS CLI since s3fs supports the same format.
Author
Owner

@nicoske commented on GitHub (Apr 21, 2021):

I did used the system wide credentials file as described in the documentation. It is stored in /etc/passwd-s3fs and I specify the path using -o passwd_file=/etc/passwd-s3fs. I used the format ACCESS_KEY_ID:SECRET_ACCESS_KEY. I already tripled checked the key and I used the same in s3cmd with success.

I just did another test by removing the option flag to that file and use $HOME/.aws/credentials with the following format:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

I have the same result:

curl.cpp:RequestPerform(2324): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message></Error>

I have a call with NetApp this morning, I'll keep you updated.

<!-- gh-comment-id:823826325 --> @nicoske commented on GitHub (Apr 21, 2021): I did used the system wide credentials file as described in the documentation. It is stored in `/etc/passwd-s3fs` and I specify the path using `-o passwd_file=/etc/passwd-s3fs`. I used the format ACCESS_KEY_ID:SECRET_ACCESS_KEY. I already tripled checked the key and I used the same in s3cmd with success. I just did another test by removing the option flag to that file and use `$HOME/.aws/credentials` with the following format: ``` [default] aws_access_key_id=AKIAIOSFODNN7EXAMPLE aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` I have the same result: `curl.cpp:RequestPerform(2324): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?><Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided.</Message></Error>` I have a call with NetApp this morning, I'll keep you updated.
Author
Owner

@gaul commented on GitHub (Apr 21, 2021):

NetApp has a tutorial on using s3fs with their product:

https://cloud.netapp.com/blog/amazon-s3-as-a-file-system

Hopefully they can offer you some pointers since I lack access to this hardware. It might help to test with AWS or Minio locally to have some comparison point. I strongly suspect that it is just misconfiguration of credentials though.

<!-- gh-comment-id:824011129 --> @gaul commented on GitHub (Apr 21, 2021): NetApp has a tutorial on using s3fs with their product: https://cloud.netapp.com/blog/amazon-s3-as-a-file-system Hopefully they can offer you some pointers since I lack access to this hardware. It might help to test with AWS or Minio locally to have some comparison point. I strongly suspect that it is just misconfiguration of credentials though.
Author
Owner

@nicoske commented on GitHub (Apr 22, 2021):

Hello,

This documentation refers to the usage of AWS S3 on any Linux system, it is not about using their S3 implementation unfortunately.

I just gave a try with the original awscli, and it works using the same credentials in $HOME/.aws/credentials than what I used for s3fs-fuse and fails:

$ aws s3 ls bucket-1 --endpoint=https://xxxx --no-verify-ssl
/usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2021-03-25 13:56:31   63447876 19774-sitevmlabo1cam-1615801202111-1615801455172.mp4

I'm going to send NetApp all the log files of the different tools I used but I suspect that there's something wrong with signature calculation by s3fs-fuse for a unknown reason..

<!-- gh-comment-id:824754499 --> @nicoske commented on GitHub (Apr 22, 2021): Hello, This documentation refers to the usage of AWS S3 on any Linux system, it is not about using their S3 implementation unfortunately. I just gave a try with the original awscli, and it works using the same credentials in `$HOME/.aws/credentials` than what I used for s3fs-fuse and fails: ``` $ aws s3 ls bucket-1 --endpoint=https://xxxx --no-verify-ssl /usr/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning) 2021-03-25 13:56:31 63447876 19774-sitevmlabo1cam-1615801202111-1615801455172.mp4 ``` I'm going to send NetApp all the log files of the different tools I used but I suspect that there's something wrong with signature calculation by s3fs-fuse for a unknown reason..
Author
Owner

@nicoske commented on GitHub (May 6, 2021):

Hi @gaul
I received some feedback from NetApp. Could it be that s3fs cannot handle keys larger than 123 characters ? By default they use a 128 characters key.

<!-- gh-comment-id:833332700 --> @nicoske commented on GitHub (May 6, 2021): Hi @gaul I received some feedback from NetApp. Could it be that s3fs cannot handle keys larger than 123 characters ? By default they use a 128 characters key.
Author
Owner

@gaul commented on GitHub (May 6, 2021):

I received some feedback from NetApp. Could it be that s3fs cannot handle keys larger than 123 characters ? By default they use a 128 characters key.

Unexpectedly, this was true -- please thank your NetApp representative for the exact cause! I wonder why they didn't submit their own PR but in the end there is a simple fix.

<!-- gh-comment-id:833383809 --> @gaul commented on GitHub (May 6, 2021): > I received some feedback from NetApp. Could it be that s3fs cannot handle keys larger than 123 characters ? By default they use a 128 characters key. Unexpectedly, this was true -- please thank your NetApp representative for the exact cause! I wonder why they didn't submit their own PR but in the end there is a simple fix.
Author
Owner

@nicoske commented on GitHub (May 6, 2021):

Thanks for the quick reaction @gaul ! I just compiled secret-key-length branch and I confirm I can mount successfully !!

<!-- gh-comment-id:833405896 --> @nicoske commented on GitHub (May 6, 2021): Thanks for the quick reaction @gaul ! I just compiled secret-key-length branch and I confirm I can mount successfully !!
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#853
No description provided.