[GH-ISSUE #1040] Input/output error on read or write only for buckets in different AWS account #573

Closed
opened 2026-03-04 01:46:50 +03:00 by kerem · 6 comments
Owner

Originally created by @shlha on GitHub (Jun 15, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1040

Bucket name: shehrtest
Mount Point: /mnt/backup

Version of s3fs being used (s3fs --version)

V1.85

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Version : 2.9.2

Kernel information (uname -r)

4.14.106-97.85.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

s3fs command line used, if applicable

Mounts bucket:
sudo s3fs -o use_path_request_style shehrtest:/data /mnt/backup -o allow_other,dbglevel=dbg,iam_role=auto

or

Mounts bucket: (If I use eu-west-2 in url or endpoint bucket is not mounted)
sudo s3fs shehrtest:/data /mnt/backup -o allow_other,dbglevel=dbg,iam_role=auto,url=https://s3-us-west-2.amazonaws.com,endpoint=us-west-2

/etc/fstab entry, if applicable

s3fs#shehrtest:/data /mnt/backup fuse _netdev,allow_other,dbglevel=dbg,iam_role=auto,uid=mssql,gid=mssql 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages
check a bucket.
Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL is https://s3.amazonaws.com/shehrtest/data/

	Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_check_service(3828): Failed to connect region 'us-east-1'(default), so retry to connect region 'eu-west-2'.

check a bucket.
Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL is https://s3-eu-west-2.amazonaws.com/shehrtest/data/

	Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: curl.cpp:RequestPerform(2252): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>E4A065D2A2F7BAF4</RequestId><HostId>CDJ+eX6aBQqLKZ9nakCTchkQoAsTEynJF1QwHOq0k4Esw2sNG1O3suMeoawgUdfbwFjmYx9Zy4k=</HostId></Error>
	Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: curl.cpp:CheckBucket(3284): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>E4A065D2A2F7BAF4</RequestId><HostId>CDJ+eX6aBQqLKZ9nakCTchkQoAsTEynJF1QwHOq0k4Esw2sNG1O3suMeoawgUdfbwFjmYx9Zy4k=</HostId></Error>
	Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_check_service(3849): Failed to connect by sigv4, so retry to connect by signature version 2.

Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: check a bucket.
Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL is https://s3-eu-west-2.amazonaws.com/shehrtest/data/
Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL changed is https://shehrtest.s3-eu-west-2.amazonaws.com/data/
Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: curl.cpp:RequestPerform(2248): HTTP response code 400, returning EIO. Body Text: #012InvalidRequestThe authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.8903040BC53DB749ZCEvAFqt8/CIevbx9/6/8Fs8bnVqYIzz5/TsBGIx10+pCuEqrC8yXcGczuYaBTLmFN9LOQmPVPA=
Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: curl.cpp:CheckBucket(3284): Check bucket failed, S3 response: #012InvalidRequestThe authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.8903040BC53DB749ZCEvAFqt8/CIevbx9/6/8Fs8bnVqYIzz5/TsBGIx10+pCuEqrC8yXcGczuYaBTLmFN9LOQmPVPA=
Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_check_service(3861): Bad Request(host=https://s3-eu-west-2.amazonaws.com) - result of checking service.
Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_exit_fuseloop(3440): Exiting FUSE event loop due to errors
Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: destroy

Details about issue

This is my first ever issue raised. I have spend hours attempting to resolve it. Have created multiple buckets, play around in all possible combinations but issue still remains there.

Either due to version of s3fs or some other reason bucket is not readable via s3fs from a different AWS account. Bucket mounts successfully but generates this error upon reading:
ls: reading directory .: Input/output error

Happens only for buckets in different AWS account. Cross account access is being achieved via attached EC2 role. If i run # aws s3 ls s3://shehrtest from EC2 instance I can read/write to it with no issues.

Originally created by @shlha on GitHub (Jun 15, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1040 Bucket name: shehrtest Mount Point: /mnt/backup #### Version of s3fs being used (s3fs --version) V1.85 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Version : 2.9.2 #### Kernel information (uname -r) 4.14.106-97.85.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" #### s3fs command line used, if applicable Mounts bucket: sudo s3fs -o use_path_request_style shehrtest:/data /mnt/backup -o allow_other,dbglevel=dbg,iam_role=auto or Mounts bucket: (If I use eu-west-2 in url or endpoint bucket is not mounted) sudo s3fs shehrtest:/data /mnt/backup -o allow_other,dbglevel=dbg,iam_role=auto,url=https://s3-us-west-2.amazonaws.com,endpoint=us-west-2 #### /etc/fstab entry, if applicable s3fs#shehrtest:/data /mnt/backup fuse _netdev,allow_other,dbglevel=dbg,iam_role=auto,uid=mssql,gid=mssql 0 0 #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ check a bucket. Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL is https://s3.amazonaws.com/shehrtest/data/ Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_check_service(3828): Failed to connect region 'us-east-1'(default), so retry to connect region 'eu-west-2'. check a bucket. Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL is https://s3-eu-west-2.amazonaws.com/shehrtest/data/ Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: curl.cpp:RequestPerform(2252): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>E4A065D2A2F7BAF4</RequestId><HostId>CDJ+eX6aBQqLKZ9nakCTchkQoAsTEynJF1QwHOq0k4Esw2sNG1O3suMeoawgUdfbwFjmYx9Zy4k=</HostId></Error> Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: curl.cpp:CheckBucket(3284): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>E4A065D2A2F7BAF4</RequestId><HostId>CDJ+eX6aBQqLKZ9nakCTchkQoAsTEynJF1QwHOq0k4Esw2sNG1O3suMeoawgUdfbwFjmYx9Zy4k=</HostId></Error> Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_check_service(3849): Failed to connect by sigv4, so retry to connect by signature version 2. Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: check a bucket. Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL is https://s3-eu-west-2.amazonaws.com/shehrtest/data/ Jun 14 21:45:50 ip-10-20-7-212 s3fs[26606]: URL changed is https://shehrtest.s3-eu-west-2.amazonaws.com/data/ Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: curl.cpp:RequestPerform(2248): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>8903040BC53DB749</RequestId><HostId>ZCEvAFqt8/CIevbx9/6/8Fs8bnVqYIzz5/TsBGIx10+pCuEqrC8yXcGczuYaBTLmFN9LOQmPVPA=</HostId></Error> Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: curl.cpp:CheckBucket(3284): Check bucket failed, S3 response: <?xml version="1.0" encoding="UTF-8"?>#012<Error><Code>InvalidRequest</Code><Message>The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.</Message><RequestId>8903040BC53DB749</RequestId><HostId>ZCEvAFqt8/CIevbx9/6/8Fs8bnVqYIzz5/TsBGIx10+pCuEqrC8yXcGczuYaBTLmFN9LOQmPVPA=</HostId></Error> Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_check_service(3861): Bad Request(host=https://s3-eu-west-2.amazonaws.com) - result of checking service. Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: s3fs.cpp:s3fs_exit_fuseloop(3440): Exiting FUSE event loop due to errors Jun 14 21:45:51 ip-10-20-7-212 s3fs[26606]: destroy ### Details about issue This is my first ever issue raised. I have spend hours attempting to resolve it. Have created multiple buckets, play around in all possible combinations but issue still remains there. Either due to version of s3fs or some other reason bucket is not readable via s3fs from a different AWS account. Bucket mounts successfully but generates this error upon reading: ls: reading directory .: Input/output error Happens only for buckets in different AWS account. Cross account access is being achieved via attached EC2 role. If i run # aws s3 ls s3://shehrtest from EC2 instance I can read/write to it with no issues.
kerem 2026-03-04 01:46:50 +03:00
  • closed this issue
  • added the
    need info
    label
Author
Owner

@ivan-ayala-mx commented on GitHub (Jun 21, 2019):

We have a similar issue, we have some aws bucket mounted and during operations one fails... then when we do a "ls" we get this:

ls: reading directory .: Input/output error
total 0

We need to umount and mount, and sometimes we need to reboot.

We are using 1.85 version.

Please, provide help on this one. I saw one ticket closed for this similar issue https://github.com/s3fs-fuse/s3fs-fuse/issues/197

<!-- gh-comment-id:504566966 --> @ivan-ayala-mx commented on GitHub (Jun 21, 2019): We have a similar issue, we have some aws bucket mounted and during operations one fails... then when we do a "ls" we get this: `ls: reading directory .: Input/output error` `total 0` We need to umount and mount, and sometimes we need to reboot. We are using 1.85 version. Please, provide help on this one. I saw one ticket closed for this similar issue https://github.com/s3fs-fuse/s3fs-fuse/issues/197
Author
Owner

@shlha commented on GitHub (Jun 22, 2019):

For myself, if i mount/umnount there is no change. There are many posts which suggest using url,endpoints and umask options but nothing seems to work. I have literally spent hours trying to find a fix.

I am starting to give up on using s3fs.

<!-- gh-comment-id:504654349 --> @shlha commented on GitHub (Jun 22, 2019): For myself, if i mount/umnount there is no change. There are many posts which suggest using url,endpoints and umask options but nothing seems to work. I have literally spent hours trying to find a fix. I am starting to give up on using s3fs.
Author
Owner

@gaul commented on GitHub (Jul 3, 2019):

@mcon-ivan Could you attach your debug logs via -d -d -f -o f2 -o curldbg flags?

<!-- gh-comment-id:508289614 --> @gaul commented on GitHub (Jul 3, 2019): @mcon-ivan Could you attach your debug logs via `-d -d -f -o f2 -o curldbg` flags?
Author
Owner

@shlha commented on GitHub (Jul 4, 2019):

Thank you for getting back, I have destroyed my test environment, Sorry

<!-- gh-comment-id:508556299 --> @shlha commented on GitHub (Jul 4, 2019): Thank you for getting back, I have destroyed my test environment, Sorry
Author
Owner

@ivan-ayala-mx commented on GitHub (Jul 4, 2019):

@mcon-ivan Could you attach your debug logs via -d -d -f -o f2 -o curldbg flags?

Thanks, we decided to use another way. This was impacting us a lot :D

I hope this will be fixed

thanks

<!-- gh-comment-id:508559518 --> @ivan-ayala-mx commented on GitHub (Jul 4, 2019): > @mcon-ivan Could you attach your debug logs via `-d -d -f -o f2 -o curldbg` flags? Thanks, we decided to use another way. This was impacting us a lot :D I hope this will be fixed thanks
Author
Owner

@gaul commented on GitHub (Feb 3, 2020):

Closing due to inactivity. Please reopen if symptoms persist.

<!-- gh-comment-id:581289482 --> @gaul commented on GitHub (Feb 3, 2020): Closing due to inactivity. Please reopen if symptoms persist.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#573
No description provided.