mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 13:26:00 +03:00
[GH-ISSUE #161] GovCloud Redirection Is Broken #95
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#95
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @wrstone on GitHub (Mar 31, 2015).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/161
Some recent code change (that didn't change the S3FS version from 1.78) has broken GovCloud redirection. Specifically:
We have a number of Ubuntu GovCloud instances. We use a standardized document to pull S3FS and compile it when installing to a new instance.
We compile and install with the following procedure:
Install S3FS
Install Development Packages
Compiling S3FS requires a number of development packages to be installed. To do so, as a non-root user, enter the following command.
Be aware that line breaks are a function of the word processor. The command should be entered on a single line:
sudo apt-get install build-essential git libfuse-dev libcurl4-openssl-dev libxml2-dev mime-support automake libtool pkg-config libssl-dev
Obtain S3FS Source Code
To compile from source, you must obtain the S3FS source code using git to clone a copy from GitHub.
To do so, enter the following command:
git clone https://github.com/s3fs-fuse/s3fs-fuse
Compile the Source Code
To compile the S3FS source code, as a non-root user, enter the commands:
cd s3fs-fuse/
./autogen.sh
./configure --prefix=/usr --with-openssl
make
Install the Compiled Binary Executable
To install the compiled executable file, enter the command:
sudo make install
Test the Installation
To test that S3FS was installed correctly, enter the following command:
This should return text similar to:
Amazon Simple Storage Service File System V1.78 with OpenSSL
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
This procedure has not changed since 16 January 2015 when I wrote it. Over the last several months, we have spun up multiple instances in GovCloud using this procedure.
When using a binary compiled on 16 January, we can successfully mount a GovCloud bucket via redirection. I.e. our standard /etc/fstab entry:
s3fs#prd.project.files /java_data/s3projects fuse allow_other,host=http://s3-us-gov-west-1.amazonaws.com 0 0
When using a binary compiled this morning, this mount fails. Specifically:
ubuntu@govs3fstest:~$ sudo s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o endpoint=us-gov-west-1 -o passwd_file=/etc/passwd-s3fs uat.project.files /mnt2
set_moutpoint_attribute(3530): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2713): init
s3fs_check_service(3070): check services.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 404
RequestPerform(1624): HTTP response code 404 was returned, returning ENOENT
CheckBucket(2590): Check bucket failed, S3 response:
NoSuchBucketThe specified bucket does not existuat.project.files.s3.amazonaws.comD38F80FDA23C32ABOlwju4mzZcyQJL8RWwa+4YeSzdM+NUtI+TaYezvAKEEIKc/Em18cdPACYYIPDyJQs3fs: bucket not found
And:
ubuntu@govs3fstest:~$ sudo s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o passwd_file=/etc/passwd-s3fs uat.project.files /mnt2
set_moutpoint_attribute(3530): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2713): init
s3fs_check_service(3070): check services.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 400
RequestPerform(1614): HTTP response code 400 was returned, returing EIO.
CheckBucket(2590): Check bucket failed, S3 response:
AuthorizationHeaderMalformedThe authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-gov-west-1'us-gov-west-140C446F54F56D3E7ERxUtFprPYGv7rgJPFEvpMFHWA4rlqDrO9xYu6PQT8PxZQN3Xs3OVoXILPwuk3GYs3fs_check_service(3103): Could not connect wrong region us-east-1, so retry to connect region us-gov-west-1.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 404
RequestPerform(1624): HTTP response code 404 was returned, returning ENOENT
CheckBucket(2590): Check bucket failed, S3 response:
NoSuchBucketThe specified bucket does not existuat.project.files.s3.amazonaws.com800D0F152C6D7F28IHmeo4jc0x5tStIF3kmYGKpnvJC1odFITl+x/58sbLOIQ9iGX061qQ/Zq16csOYts3fs: bucket not found
Clearly, with the current code, it is not correctly redirecting to the GovCloud S3 platform in spite of being explicitly told to redirect there.
Interestingly, there is a significant difference in the compiled binaries. In the following listing, the today-compiled binary is the ".old" file. The plain "s3fs" is a January compile that was copied from a production system.
root@govs3fs:/usr/bin# ls -al s3fs*
-rwxr-xr-x 1 root root 3592580 Mar 31 14:12 s3fs
-rwxr-xr-x 1 root root 4020379 Mar 31 13:57 s3fs.org
The first (January-compiled) binary works. The second (today-compiled) does not.
I would be happy to provide additional information if it's needed.
@kahing commented on GitHub (Mar 31, 2015):
How does one go about getting a govcloud S3 credential to test this?
@wrstone commented on GitHub (Mar 31, 2015):
Well, in my company's case, we have a GovCloud account that includes GovCloud S3 buckets. We're paying for it, so I'm not sure it's possible unless you fork over some cash.
However, I would certainly volunteer to spin up a test instance and perform whatever tasks might be necessary to test S3FS. It directly impacts my company: without this functionality, we are literally dead in the water.
So I'm happy to do any debugging or compiling, or anything you like. :)
@kahing commented on GitHub (Mar 31, 2015):
I misread the signup instruction and thought that I had to have a .gov association. Looks like there's some screening process before I can obtain an account and I am not sure what the delay is. I can take a look at the code to see if I can spot anything obvious.
You can speed up the process if you could give me a read only IAM role for a dummy bucket :-)
@wrstone commented on GitHub (Mar 31, 2015):
Sadly, I can't do that. Our company ITSec people would raise holy hell if we opened anything in our infrastructure to non-employees.
As I say, everything up to that I'm happy to do on our end. I actually have a test instance already set up, as it was where I discovered that the old binary works where the new one doesn't.
@kahing commented on GitHub (Mar 31, 2015):
Could you try to use -o use_path_request_style? I wonder if the v4 signing broke this, so maybe also try with -o sigv2?
@kahing commented on GitHub (Mar 31, 2015):
also, could you provide the debug log from the old s3fs binary that worked?
@wrstone commented on GitHub (Mar 31, 2015):
-o sigv2 worked.
Here's -o user_path_request_style. I just had something come up that's going to interrupt me doing anything with this for the rest of the day, but I'll get that debug info as soon as I can.
[ubuntu@govs3fs ~]$ sudo s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o passwd_file=/etc/passwd-s3fs -o use_path_request_style uat.project.files /mnt2
set_moutpoint_attribute(3530): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2713): init
s3fs_check_service(3070): check services.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://s3-us-gov-west-1.amazonaws.com/uat.project.files/
RequestPerform(1600): HTTP response code 400
RequestPerform(1614): HTTP response code 400 was returned, returing EIO.
CheckBucket(2590): Check bucket failed, S3 response:
AuthorizationHeaderMalformedThe authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-gov-west-1'us-gov-west-1E42BD440D8BC41C13qk5iNX2ihvZ+ct5LoRG3FT5PeG7NedMikk48c2JHbtNVEbi+3tk46PQkNxxl5uCs3fs_check_service(3103): Could not connect wrong region us-east-1, so retry to connect region us-gov-west-1.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://s3-us-gov-west-1.amazonaws.com/uat.project.files/
RequestPerform(1600): HTTP response code 403
RequestPerform(1619): HTTP response code 403 was returned, returning EPERM
CheckBucket(2590): Check bucket failed, S3 response:
SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.AKIALK5PIY5CLWXSVAFQAWS4-HMAC-SHA25620150331T175107Z
20150331/us-gov-west-1/s3/aws4_request
6254fceab7e89d269d3c0038fc78344d1a90f94dd7630c7b4c91c8876cf88ca9f5e260d52341c2897bad52b5bd4f9ffa13e1dd0ed977bce4a75eff32a027c6f441 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 33 33 31 54 31 37 35 31 30 37 5a 0a 32 30 31 35 30 33 33 31 2f 75 73 2d 67 6f 76 2d 77 65 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 36 32 35 34 66 63 65 61 62 37 65 38 39 64 32 36 39 64 33 63 30 30 33 38 66 63 37 38 33 34 34 64 31 61 39 30 66 39 34 64 64 37 36 33 30 63 37 62 34 63 39 31 63 38 38 37 36 63 66 38 38 63 61 39GET
/uat.project.files/
host:uat.project.files.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150331T175107Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85547 45 54 0a 2f 75 61 74 2e 70 72 6f 6a 65 63 74 2e 66 69 6c 65 73 2f 0a 0a 68 6f 73 74 3a 75 61 74 2e 70 72 6f 6a 65 63 74 2e 66 69 6c 65 73 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 30 33 33 31 54 31 37 35 31 30 37 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 3556EF0D005EFF52A1r5HHnwkwWl1OYWBiGHVHbgSiVSa0fNAncKgMWJ93CKxcDnhKGn7mTXWMASY+xnzE
s3fs: Failed to access bucket.
@kahing commented on GitHub (Mar 31, 2015):
ok good that you have a workaround for now.
@wrstone commented on GitHub (Apr 2, 2015):
Sorry it's taken so long to get back -- as usual, work and personal life were hectic.
Here's what I get from the old binary. I'm not sure what to look for in regards to other debugging output. If you can tell me where to look, I'd be happy to post it.
[root@govs3fs /usr/bin]# sudo s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o passwd_file=/etc/passwd-s3fs uat.project.files /mnt2
set_moutpoint_attribute(3389): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2660): init
s3fs_check_service(2978): check services.
CheckBucket(2366): check a bucket.
RequestPerform(1571): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1587): HTTP response code 200
At that point, the S3 bucket appears mounted. S3FS will sit at this screen until killed with a CTRL-C.
@kahing commented on GitHub (Apr 3, 2015):
The problem is probably in the v4 signer which is new. I will get to this but the process of getting a GovCloud credential is slower than I had expected.
@kahing commented on GitHub (Apr 9, 2015):
the pull request should fix the bug (did a simple read only test with my gov cloud credential), @wrstone could you verify?
@wrstone commented on GitHub (Apr 10, 2015):
Unfortunately, that doesn't seem to solve the problem. This is from a binary compiled from code pulled a few minutes ago.
The console output follows. tl;dr: it still seems to redirect to Public S3 rather than GovCloud.
The various command-line output:
[root@govs3fs /]# s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o passwd_file=/etc/passwd-s3fs uat.project.files /mnt2
set_moutpoint_attribute(3530): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2713): init
s3fs_check_service(3070): check services.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 400
RequestPerform(1614): HTTP response code 400 was returned, returing EIO.
CheckBucket(2590): Check bucket failed, S3 response:
AuthorizationHeaderMalformedThe authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-gov-west-1'us-gov-west-1FFBB5B4967FDA36FI0VrgOmFxtiPdC5AZGJvxlWLJgz/KKV2gpKE07fe6rvD4dqHV0b+/kBcgL0BrCFSs3fs_check_service(3103): Could not connect wrong region us-east-1, so retry to connect region us-gov-west-1.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 404
RequestPerform(1624): HTTP response code 404 was returned, returning ENOENT
CheckBucket(2590): Check bucket failed, S3 response:
NoSuchBucketThe specified bucket does not existuat.project.files.s3.amazonaws.com219BBEC7668C09034TqeU8MbMCDGweOsUSPspon1r7SbxMSROLKx7CFBdJFzes2Brqq30KK3+vLZUcLHs3fs: bucket not found
And:
[root@govs3fs /]# s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o passwd_file=/etc/passwd-s3fs uat.project.files /mnt2
set_moutpoint_attribute(3530): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2713): init
s3fs_check_service(3070): check services.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 400
RequestPerform(1614): HTTP response code 400 was returned, returing EIO.
CheckBucket(2590): Check bucket failed, S3 response:
AuthorizationHeaderMalformedThe authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-gov-west-1'us-gov-west-11AE51611B55CE379uNFRIknxsYAFmG0BHEErmWcb/N2ysNyZvNt4Cm4MFEoQhiiEpcpHzxyhrHmNjGuPs3fs_check_service(3103): Could not connect wrong region us-east-1, so retry to connect region us-gov-west-1.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://uat.project.files.s3-us-gov-west-1.amazonaws.com/
RequestPerform(1600): HTTP response code 404
RequestPerform(1624): HTTP response code 404 was returned, returning ENOENT
CheckBucket(2590): Check bucket failed, S3 response:
NoSuchBucketThe specified bucket does not existuat.project.files.s3.amazonaws.com7FAF4E3ACFF32C846iQ+fPfwi3JozdKLaQ/FFM2FPfCD+YFWQgPLcwr6ZeQUVhHiACuQMfh7omhWr6oUs3fs: bucket not found
And:
[root@govs3fs /]# s3fs -f -d -o url=http://s3-us-gov-west-1.amazonaws.com -o multireq_max=5 -o allow_other -o passwd_file=/etc/passwd-s3fs -o use_path_request_style uat.project.files /mnt2
set_moutpoint_attribute(3530): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40755)
s3fs_init(2713): init
s3fs_check_service(3070): check services.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://s3-us-gov-west-1.amazonaws.com/uat.project.files/
RequestPerform(1600): HTTP response code 400
RequestPerform(1614): HTTP response code 400 was returned, returing EIO.
CheckBucket(2590): Check bucket failed, S3 response:
AuthorizationHeaderMalformedThe authorization header is malformed; the region 'us-east-1' is wrong; expecting 'us-gov-west-1'us-gov-west-1B500955D01EEC3FCHjq1SYz5D7HydjgEZS06UT36HffKUlWbqdp8KBsTXj5K8ealzYo3SsR9R27JH5k3s3fs_check_service(3103): Could not connect wrong region us-east-1, so retry to connect region us-gov-west-1.
CheckBucket(2538): check a bucket.
RequestPerform(1584): connecting to URL http://s3-us-gov-west-1.amazonaws.com/uat.project.files/
RequestPerform(1600): HTTP response code 403
RequestPerform(1619): HTTP response code 403 was returned, returning EPERM
CheckBucket(2590): Check bucket failed, S3 response:
SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.AKIALK5PIY5CLWXSVAFQAWS4-HMAC-SHA25620150410T124058Z
20150410/us-gov-west-1/s3/aws4_request
403d019318d79d39026932e18a9d00cc3ef8f9c4fb3bd127ab551cb32025c815d53da0e2f0087beded86c2ec4f949aafbc772edb4e5366565f44330e3bb33c7241 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 35 30 34 31 30 54 31 32 34 30 35 38 5a 0a 32 30 31 35 30 34 31 30 2f 75 73 2d 67 6f 76 2d 77 65 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 34 30 33 64 30 31 39 33 31 38 64 37 39 64 33 39 30 32 36 39 33 32 65 31 38 61 39 64 30 30 63 63 33 65 66 38 66 39 63 34 66 62 33 62 64 31 32 37 61 62 35 35 31 63 62 33 32 30 32 35 63 38 31 35GET
/uat.project.files/
host:uat.project.files.s3.amazonaws.com
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20150410T124058Z
host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85547 45 54 0a 2f 75 61 74 2e 70 72 6f 6a 65 63 74 2e 66 69 6c 65 73 2f 0a 0a 68 6f 73 74 3a 75 61 74 2e 70 72 6f 6a 65 63 74 2e 66 69 6c 65 73 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 35 30 34 31 30 54 31 32 34 30 35 38 5a 0a 0a 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 65 33 62 30 63 34 34 32 39 38 66 63 31 63 31 34 39 61 66 62 66 34 63 38 39 39 36 66 62 39 32 34 32 37 61 65 34 31 65 34 36 34 39 62 39 33 34 63 61 34 39 35 39 39 31 62 37 38 35 32 62 38 35 35F9FFDBE565CCFD26KIr9V4AGU6VF0UqK4ecQjs0JnJYBj+JkBwkXvaPaXTNtICr2TzrJ509ObmV8HPJ1
s3fs: Failed to access bucket.
@kahing commented on GitHub (Apr 10, 2015):
the fix hasn't been merged yet, you need this commit on top of s3fs master:
github.com/s3fs-fuse/s3fs-fuse@98af055d8b@wrstone commented on GitHub (Apr 14, 2015):
Sorry it took a while to get back to this. Unfortunately, they won't pay me to work on this full-time ... ;)
I can confirm that the new curl.cpp resolved the issue. I was able to successfully mount a bucket at the command line and via a standard /etc/fstab entry.
@adamxyzxyz commented on GitHub (Dec 16, 2020):
I realize this thread is super load, but for anyone that is looking at this now and is struggling to mount their s3 bucket to a govcloud instance...
You NEED to make sure that you point s3fs to the correct URL. If you don't include this in the command to mount the bucket, then it will not mount correctly.
Use the command below to mount:
sudo s3fs {your bucket name here} /{directory to mount in} -o passwd_file=${HOME}/.passwd-s3fs -o url=https://s3-us-gov-east-1.amazonaws.com -o use_path_request_style
INCLUDE EVERYTHING AFTER URL OR IT WON'T MOUNT.