starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00

[GH-ISSUE #1317] HTTP 200 might be an error response #704

New issue

Closed

opened 2026-03-04 01:48:03 +03:00 by kerem · 6 comments

kerem commented

2026-03-04 01:48:03 +03:00

Owner

Originally created by @KFerkPerf on GitHub (Jun 23, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1317

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

V1.86

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

V2.9.2

Kernel information (uname -r)

4.14.181-140.257.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

command result: cat /etc/os-release

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

s3fs command line used, if applicable

s3fs wab-jer-storage-stage /s3/data -o rw,allow_other,iam_role=myrole,dev,suid

/etc/fstab entry, if applicable

s3fs#wab-jer-storage-stage /s3/data fuse _netdev,allow_other,iam_role=myrole 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

Details about issue

For some commands requests might got a 200 OK response but contain error message in response body:
https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-200-internalerror/

Can't find check in code and problem seems to hit us. There is a .csv file moved to another location. The result file contains an error message:

uid1234<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>reqid</RequestId><HostId>hostid</HostId></Error>uid3456,10,126,604197,...

In case of 200 but error in body, retry like for 500 responses to be done

Originally created by @KFerkPerf on GitHub (Jun 23, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1317 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) _V1.86_ #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _V2.9.2_ #### Kernel information (uname -r) 4.14.181-140.257.amzn2.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) _command result: cat /etc/os-release_ ``` NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" ``` #### s3fs command line used, if applicable ``` s3fs wab-jer-storage-stage /s3/data -o rw,allow_other,iam_role=myrole,dev,suid ``` #### /etc/fstab entry, if applicable ``` s3fs#wab-jer-storage-stage /s3/data fuse _netdev,allow_other,iam_role=myrole 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ ``` ``` ### Details about issue For some commands requests might got a 200 OK response but contain error message in response body: https://aws.amazon.com/premiumsupport/knowledge-center/s3-resolve-200-internalerror/ Can't find check in code and problem seems to hit us. There is a .csv file moved to another location. The result file contains an error message: ``` uid1234<?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>reqid</RequestId><HostId>hostid</HostId></Error>uid3456,10,126,604197,... ``` In case of 200 but error in body, retry like for 500 responses to be done

kerem

2026-03-04 01:48:03 +03:00

closed this issue
added the
bug
label

kerem commented

2026-03-04 01:48:04 +03:00

Author

Owner

@tsmgeek commented on GitHub (Jul 3, 2020):

Does this means currently s3fs checks for 200 for a successful write wheras it can actually be a problem?

@tsmgeek commented on GitHub (Jul 3, 2020): Does this means currently s3fs checks for 200 for a successful write wheras it can actually be a problem?

kerem commented

2026-03-04 01:48:04 +03:00

Author

Owner

@KFerkPerf commented on GitHub (Jul 6, 2020):

Does this means currently s3fs checks for 200 for a successful write wheras it can actually be a problem?

Exactly, that's what also stated by amazon documentation. The response is 200 but the response body contains an error message.

@KFerkPerf commented on GitHub (Jul 6, 2020): > Does this means currently s3fs checks for 200 for a successful write wheras it can actually be a problem? Exactly, that's what also stated by amazon documentation. The response is 200 but the response body contains an error message.

kerem commented

2026-03-04 01:48:04 +03:00

Author

Owner

@tsmgeek commented on GitHub (Jul 6, 2020):

@KFerkPerf do you think this pollutes the fuse cache?
I'm seeing an odd scenario where my S3 storage has an empty file, the fwrite was successful but a process running on the same machine that reads the file immediately after write (in another process) sees all the data correctly for a period of time.

@tsmgeek commented on GitHub (Jul 6, 2020): @KFerkPerf do you think this pollutes the fuse cache? I'm seeing an odd scenario where my S3 storage has an empty file, the fwrite was successful but a process running on the same machine that reads the file immediately after write (in another process) sees all the data correctly for a period of time.

kerem commented

2026-03-04 01:48:04 +03:00

Author

Owner

@KFerkPerf commented on GitHub (Jul 7, 2020):

Not sure if this is connected as this is more about having corrupted data during reads than inconsistent view. Your description sound's more like a general problem with caching. I got a similiar problem when working with dockerized environments where mounts there where cached.

@KFerkPerf commented on GitHub (Jul 7, 2020): Not sure if this is connected as this is more about having corrupted data during reads than inconsistent view. Your description sound's more like a general problem with caching. I got a similiar problem when working with dockerized environments where mounts there where cached.

kerem commented

2026-03-04 01:48:05 +03:00

Author

Owner

@gaul commented on GitHub (Jul 7, 2020):

It seems crazy for S3 to return 200 on an actual error but I can imagine this happening when something breaks between the initial response and streaming the body. s3fs should first check the XML body and, if it does not exist, fall back to HTTP status codes. This seems like a rare condition so I am not sure if this relates to your caching symptoms.

@gaul commented on GitHub (Jul 7, 2020): It seems crazy for S3 to return 200 on an actual error but I can imagine this happening when something breaks between the initial response and streaming the body. s3fs should first check the XML body and, if it does not exist, fall back to HTTP status codes. This seems like a rare condition so I am not sure if this relates to your caching symptoms.

kerem commented

2026-03-04 01:48:05 +03:00

Author

Owner

@KFerkPerf commented on GitHub (Jul 7, 2020):

@gaul not sure what really causes this behavior. I think the body does exist always but normally should contain the data rather than the error message. So get would return following response with http code 200:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>reqid</RequestId><HostId>hostid</HostId></Error>

Checking for this error might add some overhead to the performance but I think it is necessary or at least it should be possible to switch on this check.

@KFerkPerf commented on GitHub (Jul 7, 2020): @gaul not sure what really causes this behavior. I think the body does exist always but normally should contain the data rather than the error message. So get would return following response with http code 200: ``` <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InternalError</Code><Message>We encountered an internal error. Please try again.</Message><RequestId>reqid</RequestId><HostId>hostid</HostId></Error> ``` Checking for this error might add some overhead to the performance but I think it is necessary or at least it should be possible to switch on this check.

kerem referenced this issue

2026-03-04 02:01:19 +03:00

[PR #704] [MERGED] Update README.md with details about .passwd-s3fs #1611