[GH-ISSUE #847] Problems listing buckets of S3 compatible service with virtual-host style requests #495

New issue

Closed

opened 2026-03-04 01:46:06 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:46:06 +03:00

Owner

Originally created by @ffeldhaus on GitHub (Oct 29, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/847

Additional Information

When connecting to an S3 compatible service, listing the bucket content fails.

Version of s3fs being used (s3fs --version)

[root@florianf-linux ~]# s3fs --version
Amazon Simple Storage Service File System V1.84(commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

[root@florianf-linux ~]# rpm -qi fuse
Name        : fuse
Version     : 2.9.2
Release     : 10.el7
Architecture: x86_64
Install Date: Mon 29 Oct 2018 12:43:24 PM CET
Group       : System Environment/Base
Size        : 223041
License     : GPL+
Signature   : RSA/SHA256, Wed 25 Apr 2018 01:00:05 PM CEST, Key ID 24c6a8a7f4a80eb5
Source RPM  : fuse-2.9.2-10.el7.src.rpm
Build Date  : Wed 11 Apr 2018 07:12:48 AM CEST
Build Host  : x86-01.bsys.centos.org
Relocations : (not relocatable)
Packager    : CentOS BuildSystem <http://bugs.centos.org>
Vendor      : CentOS

Kernel information (uname -r)

[root@florianf-linux ~]# uname -r
3.10.0-862.14.4.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

[root@florianf-linux ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

s3fs command line used, if applicable

 s3fs cloudmirror-1 /mnt/cloudmirror-1 -o url=https://s3.labdus.netapp.com:8082 -f -d

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

[root@florianf-linux ~]# s3fs cloudmirror-1 /mnt/cloudmirror-1 -o url=https://s3.labdus.netapp.com:8082/ -f -d
[CRT] s3fs.cpp:set_s3fs_log_level(273): change debug level from [CRT] to [INF]
[INF]     s3fs.cpp:set_mountpoint_attribute(4180): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
[INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with OpenSSL
[INF] s3fs.cpp:s3fs_check_service(3721): check services.
[INF]       curl.cpp:CheckBucket(3076): check a bucket.
[INF]       curl.cpp:prepare_url(4265): URL is https://s3.labdus.netapp.com:8082/cloudmirror-1/
[INF]       curl.cpp:prepare_url(4297): URL changed is https://cloudmirror-1.s3.labdus.netapp.com:8082/
[INF]       curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(101): url is https://s3.labdus.netapp.com:8082
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[INF] s3fs.cpp:s3fs_getattr(843): [path=/]
[INF] s3fs.cpp:s3fs_opendir(2283): [path=/][flags=100352]
[INF] s3fs.cpp:s3fs_readdir(2434): [path=/]
[INF]   s3fs.cpp:list_bucket(2477): [path=/]
[INF]       curl.cpp:ListBucketRequest(3111): [tpath=/]
[INF]       curl.cpp:prepare_url(4265): URL is https://s3.labdus.netapp.com:8082/cloudmirror-1?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:prepare_url(4297): URL changed is https://cloudmirror-1.s3.labdus.netapp.com:8082?delimiter=/&max-keys=1000&prefix=
[INF]       curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
[INF]       curl.cpp:url_to_host(101): url is https://s3.labdus.netapp.com:8082
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200
[ERR] s3fs.cpp:get_base_exp(2749): marker_xp->nodesetval is empty.
[ERR] s3fs.cpp:get_base_exp(2749): marker_xp->nodesetval is empty.
[INF] s3fs.cpp:s3fs_getattr(843): [path=/]

Details about issue

A thing which seems to be wrong is, that s3fs removes the / after the url (before ?delimiter) when using the default virtual-host style:

URL changed is https://cloudmirror-1.s3.labdus.netapp.com:8082?delimiter=/&max-keys=1000&prefix=

As the same command works when specifying -o use_path_request_style I suspect that there is something wrong with virtual-host style, probably with rewriting the URL. DNS resolution of the bucket name (e.g. cloudmirror-1.s3.labdus.netapp.com) works without problems. Interestingly virtual-host style Bucket Listing works with AWS S3.

Originally created by @ffeldhaus on GitHub (Oct 29, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/847 ### Additional Information When connecting to an S3 compatible service, listing the bucket content fails. #### Version of s3fs being used (s3fs --version) ``` [root@florianf-linux ~]# s3fs --version Amazon Simple Storage Service File System V1.84(commit:unknown) with OpenSSL ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` [root@florianf-linux ~]# rpm -qi fuse Name : fuse Version : 2.9.2 Release : 10.el7 Architecture: x86_64 Install Date: Mon 29 Oct 2018 12:43:24 PM CET Group : System Environment/Base Size : 223041 License : GPL+ Signature : RSA/SHA256, Wed 25 Apr 2018 01:00:05 PM CEST, Key ID 24c6a8a7f4a80eb5 Source RPM : fuse-2.9.2-10.el7.src.rpm Build Date : Wed 11 Apr 2018 07:12:48 AM CEST Build Host : x86-01.bsys.centos.org Relocations : (not relocatable) Packager : CentOS BuildSystem <http://bugs.centos.org> Vendor : CentOS ``` #### Kernel information (uname -r) ``` [root@florianf-linux ~]# uname -r 3.10.0-862.14.4.el7.x86_64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` [root@florianf-linux ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" ``` #### s3fs command line used, if applicable ``` s3fs cloudmirror-1 /mnt/cloudmirror-1 -o url=https://s3.labdus.netapp.com:8082 -f -d ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` [root@florianf-linux ~]# s3fs cloudmirror-1 /mnt/cloudmirror-1 -o url=https://s3.labdus.netapp.com:8082/ -f -d [CRT] s3fs.cpp:set_s3fs_log_level(273): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4180): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755) [INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with OpenSSL [INF] s3fs.cpp:s3fs_check_service(3721): check services. [INF] curl.cpp:CheckBucket(3076): check a bucket. [INF] curl.cpp:prepare_url(4265): URL is https://s3.labdus.netapp.com:8082/cloudmirror-1/ [INF] curl.cpp:prepare_url(4297): URL changed is https://cloudmirror-1.s3.labdus.netapp.com:8082/ [INF] curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(101): url is https://s3.labdus.netapp.com:8082 [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [INF] s3fs.cpp:s3fs_getattr(843): [path=/] [INF] s3fs.cpp:s3fs_opendir(2283): [path=/][flags=100352] [INF] s3fs.cpp:s3fs_readdir(2434): [path=/] [INF] s3fs.cpp:list_bucket(2477): [path=/] [INF] curl.cpp:ListBucketRequest(3111): [tpath=/] [INF] curl.cpp:prepare_url(4265): URL is https://s3.labdus.netapp.com:8082/cloudmirror-1?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:prepare_url(4297): URL changed is https://cloudmirror-1.s3.labdus.netapp.com:8082?delimiter=/&max-keys=1000&prefix= [INF] curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] [] [INF] curl.cpp:url_to_host(101): url is https://s3.labdus.netapp.com:8082 [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 [ERR] s3fs.cpp:get_base_exp(2749): marker_xp->nodesetval is empty. [ERR] s3fs.cpp:get_base_exp(2749): marker_xp->nodesetval is empty. [INF] s3fs.cpp:s3fs_getattr(843): [path=/] ``` ### Details about issue A thing which seems to be wrong is, that s3fs removes the / after the url (before `?delimiter`) when using the default virtual-host style: ``` URL changed is https://cloudmirror-1.s3.labdus.netapp.com:8082?delimiter=/&max-keys=1000&prefix= ``` As the same command works when specifying `-o use_path_request_style` I suspect that there is something wrong with virtual-host style, probably with rewriting the URL. DNS resolution of the bucket name (e.g. cloudmirror-1.s3.labdus.netapp.com) works without problems. Interestingly virtual-host style Bucket Listing works with AWS S3.

kerem closed this issue

2026-03-04 01:46:06 +03:00

kerem commented

2026-03-04 01:46:07 +03:00

Author

Owner

@gaul commented on GitHub (Jul 11, 2019):

Do virtual-host style buckets work with other tools when using your non-AWS S3 service?

@gaul commented on GitHub (Jul 11, 2019): Do virtual-host style buckets work with other tools when using your non-AWS S3 service?

kerem commented

2026-03-04 01:46:07 +03:00

Author

Owner

@ffeldhaus commented on GitHub (Jul 11, 2019):

The issue does not occur on newer releases of s3fs anymore, therefore closing the issue.

@ffeldhaus commented on GitHub (Jul 11, 2019): The issue does not occur on newer releases of s3fs anymore, therefore closing the issue.

kerem referenced this issue

2026-03-04 02:00:52 +03:00

[PR #495] [MERGED] Fix invalid V4 signature on multipart copy requests #1524