[GH-ISSUE #1184] prepare_url screws up the uri if the bucket name and endpoint prefix match #625

New issue

Closed

opened 2026-03-04 01:47:19 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:47:19 +03:00

Owner

Originally created by @bryceml on GitHub (Oct 26, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1184

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.84(commit:unknown) with GnuTLS(gcrypt)
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Version: 2.9.9-1ubuntu1

Kernel information (uname -r)

5.3.0-19-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

Ubuntu19.10

s3fs command line used, if applicable

s3fs -f /tmp/test -o url=http://s3.repo.saltstack.com -o use_path_request_style -o public_bucket=1 -o bucket=s3

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

[INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with GnuTLS(gcrypt)
[CRT] s3fs.cpp:s3fs_check_service(3780): bucket not found(host=http://s3.repo.saltstack.com) - result of checking service.

Details about issue

What I can determine from the issue, is that if the beginning of the url before the first period matches the bucket name, it screws up calculating the bucket.

What works:
s3fs -f /tmp/test -o url=http://dezb4fersljmf.cloudfront.net -o use_path_request_style -o public_bucket=1 -o bucket=s3

If I do a wireshark packet capture, I see it is checking /s3/ as the uri

What doesn't work:
s3fs -f /tmp/test -o url=http://s3.repo.saltstack.com -o use_path_request_style -o public_bucket=1 -o bucket=s3

If I do a wireshark packet capture, I see it is trying to fetch /s3//s3/.repo.saltstack.com/s3/

I re-ran it with debug level info and got the following:

[CRT] s3fs.cpp:set_s3fs_log_level(273): change debug level from [CRT] to [INF] 
[INF]     s3fs.cpp:set_mountpoint_attribute(4180): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775)
[INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with GnuTLS(gcrypt)
[INF] s3fs.cpp:s3fs_check_service(3721): check services.
[INF]       curl.cpp:CheckBucket(3076): check a bucket.
[INF]       curl.cpp:prepare_url(4265): URL is http://s3.repo.saltstack.com/s3/
[INF]       curl.cpp:prepare_url(4297): URL changed is http://s3.repo.saltstack.com/s3//s3/.repo.saltstack.com/s3/
[INF]       curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(101): url is http://s3.repo.saltstack.com
[INF]       curl.cpp:RequestPerform(2082): HTTP response code 404 was returned, returning ENOENT
[ERR] curl.cpp:CheckBucket(3104): Check bucket failed, S3 response: <!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
</head>
<body>
  <div id="listing"></div>
  <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
  <script type="text/javascript">
    var BUCKET_URL = '/s3';
    var BUCKET_WEBSITE_URL = '';
  </script>
  <script type="text/javascript" src="/list.js"></script>
</body>
</html>

[CRT] s3fs.cpp:s3fs_check_service(3780): bucket not found(host=http://s3.repo.saltstack.com) - result of checking service.
[ERR] s3fs.cpp:s3fs_exit_fuseloop(3369): Exiting FUSE event loop due to errors

[INF] s3fs.cpp:s3fs_destroy(3428): destroy

Here is the output of the info log when the host name doesn't match:

[CRT] s3fs.cpp:set_s3fs_log_level(273): change debug level from [CRT] to [INF] 
[INF]     s3fs.cpp:set_mountpoint_attribute(4180): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775)
[INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with GnuTLS(gcrypt)
[INF] s3fs.cpp:s3fs_check_service(3721): check services.
[INF]       curl.cpp:CheckBucket(3076): check a bucket.
[INF]       curl.cpp:prepare_url(4265): URL is http://dezb4fersljmf.cloudfront.net/s3/
[INF]       curl.cpp:prepare_url(4297): URL changed is http://dezb4fersljmf.cloudfront.net/s3/
[INF]       curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [] []
[INF]       curl.cpp:url_to_host(101): url is http://dezb4fersljmf.cloudfront.net
[INF]       curl.cpp:RequestPerform(2062): HTTP response code 200

Originally created by @bryceml on GitHub (Oct 26, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1184 ### Additional Information #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.84(commit:unknown) with GnuTLS(gcrypt) Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Version: 2.9.9-1ubuntu1 #### Kernel information (uname -r) 5.3.0-19-generic #### GNU/Linux Distribution, if applicable (cat /etc/os-release) Ubuntu19.10 #### s3fs command line used, if applicable ``` s3fs -f /tmp/test -o url=http://s3.repo.saltstack.com -o use_path_request_style -o public_bucket=1 -o bucket=s3 ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` [INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with GnuTLS(gcrypt) [CRT] s3fs.cpp:s3fs_check_service(3780): bucket not found(host=http://s3.repo.saltstack.com) - result of checking service. ``` ### Details about issue What I can determine from the issue, is that if the beginning of the url before the first period matches the bucket name, it screws up calculating the bucket. What works: s3fs -f /tmp/test -o url=http://dezb4fersljmf.cloudfront.net -o use_path_request_style -o public_bucket=1 -o bucket=s3 If I do a wireshark packet capture, I see it is checking /s3/ as the uri What doesn't work: s3fs -f /tmp/test -o url=http://s3.repo.saltstack.com -o use_path_request_style -o public_bucket=1 -o bucket=s3 If I do a wireshark packet capture, I see it is trying to fetch /s3//s3/.repo.saltstack.com/s3/ I re-ran it with debug level info and got the following: ``` [CRT] s3fs.cpp:set_s3fs_log_level(273): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4180): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775) [INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with GnuTLS(gcrypt) [INF] s3fs.cpp:s3fs_check_service(3721): check services. [INF] curl.cpp:CheckBucket(3076): check a bucket. [INF] curl.cpp:prepare_url(4265): URL is http://s3.repo.saltstack.com/s3/ [INF] curl.cpp:prepare_url(4297): URL changed is http://s3.repo.saltstack.com/s3//s3/.repo.saltstack.com/s3/ [INF] curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(101): url is http://s3.repo.saltstack.com [INF] curl.cpp:RequestPerform(2082): HTTP response code 404 was returned, returning ENOENT [ERR] curl.cpp:CheckBucket(3104): Check bucket failed, S3 response: <!DOCTYPE html> <html> <head> <meta charset="utf-8"> </head> <body> <div id="listing"></div> <script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script> <script type="text/javascript"> var BUCKET_URL = '/s3'; var BUCKET_WEBSITE_URL = ''; </script> <script type="text/javascript" src="/list.js"></script> </body> </html> [CRT] s3fs.cpp:s3fs_check_service(3780): bucket not found(host=http://s3.repo.saltstack.com) - result of checking service. [ERR] s3fs.cpp:s3fs_exit_fuseloop(3369): Exiting FUSE event loop due to errors [INF] s3fs.cpp:s3fs_destroy(3428): destroy ``` Here is the output of the info log when the host name doesn't match: ``` [CRT] s3fs.cpp:set_s3fs_log_level(273): change debug level from [CRT] to [INF] [INF] s3fs.cpp:set_mountpoint_attribute(4180): PROC(uid=1000, gid=1000) - MountPoint(uid=1000, gid=1000, mode=40775) [INF] s3fs.cpp:s3fs_init(3379): init v1.84(commit:unknown) with GnuTLS(gcrypt) [INF] s3fs.cpp:s3fs_check_service(3721): check services. [INF] curl.cpp:CheckBucket(3076): check a bucket. [INF] curl.cpp:prepare_url(4265): URL is http://dezb4fersljmf.cloudfront.net/s3/ [INF] curl.cpp:prepare_url(4297): URL changed is http://dezb4fersljmf.cloudfront.net/s3/ [INF] curl.cpp:insertV4Headers(2408): computing signature [GET] [/] [] [] [INF] curl.cpp:url_to_host(101): url is http://dezb4fersljmf.cloudfront.net [INF] curl.cpp:RequestPerform(2062): HTTP response code 200 ```