mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 05:16:00 +03:00
[GH-ISSUE #581] Input/output error on read. But write is success. #329
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#329
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @umamaheswararao on GitHub (May 5, 2017).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/581
Additional Information
The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Version of s3fs being used (s3fs --version)
Used master branch.
System information (uname -a)
Linux WORK 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Distro (cat /etc/issue)
Ubuntu 14.04.2 LTS \n \l
s3fs command line used (if applicable)
if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages
May 5 14:54:27 WORK s3fs[9416]: s3fs.cpp:s3fs_init(3377): init v1.80(commit:f02b1bc) with OpenSSL
May 5 14:54:27 WORK s3fs[9416]: s3fs.cpp:s3fs_check_service(3776): Could not connect wrong region us-east-1, so retry to connect region us-west-1.
Details about issue
I was able to mount and able to create files successfully from the mounted directory.
When I try to read the files It thorws Input/outpur error.
WORK:
/mydev/LocalHadoop/s3mount$ touch testfile/mydev/LocalHadoop/s3mount$ vi testfileWORK:
WORK:
/mydev/LocalHadoop/s3mount$ cat testfile/mydev/LocalHadoop/s3mount$cat: testfile: Input/output error
WORK:
@umamaheswararao commented on GitHub (May 6, 2017):
When I enable debug mode, I can see following trace....
Looks like some authorization issue: HTTP/1.1 403 Forbidden
[INF] curl.cpp:RequestPerform(1940): HTTP response code 200
[DBG] curl.cpp:ReturnHandler(306): Return handler to pool: 31
[INF] cache.cpp:AddStat(356): add stat cache entry[path=/testfile]
[DBG] cache.cpp:GetStat(276): stat cache hit [path=/testfile][time=15836537.570130182][hit count=0]
[DBG] s3fs.cpp:get_object_attribute(408): [path=/testfile]
[DBG] cache.cpp:GetStat(276): stat cache hit [path=/testfile][time=15836537.570130182][hit count=1]
[DBG] fdcache.cpp:Open(2016): [path=/testfile][size=7][time=1494026969]
[DBG] fdcache.cpp:Open(790): [path=/testfile][fd=-1][size=7][time=1494026969]
[INF] fdcache.cpp:SetMtime(1015): [path=/testfile][fd=9][time=1494026969]
open[9] flags: 0x8000 /testfile
unique: 35, success, outsize: 32
unique: 36, opcode: READ (15), nodeid: 4, insize: 80, pid: 12649
read[9] 4096 bytes from 0 flags: 0x8000
[DBG] s3fs.cpp:s3fs_read(2120): [path=/testfile][size=4096][offset=0][fd=9]
[DBG] fdcache.cpp:ExistOpen(2082): [path=/testfile][fd=9][ignore_existfd=false]
[DBG] fdcache.cpp:Open(2016): [path=/testfile][size=-1][time=-1]
[DBG] fdcache.cpp:Open(790): [path=/testfile][fd=9][size=-1][time=-1]
[DBG] fdcache.cpp:Dup(730): [path=/testfile][fd=9][refcnt=2]
[DBG] fdcache.cpp:Read(1550): [path=/testfile][fd=9][offset=0][size=4096]
[DBG] fdcache.cpp:Load(1123): [path=/testfile][fd=9][offset=0][size=4096]
[INF] curl.cpp:GetObjectRequest(2865): [tpath=/testfile][start=0][size=7]
[DBG] s3fs.cpp:get_object_attribute(408): [path=/testfile]
[DBG] cache.cpp:GetStat(276): stat cache hit [path=/testfile][time=15836537.570130182][hit count=2]
[INF] curl.cpp:PreGetObjectRequest(2794): [tpath=/testfile][start=0][size=7]
[DBG] curl.cpp:GetHandler(283): Get handler from pool: 31
[INF] curl.cpp:prepare_url(4162): URL is http://s3-us-west-1.amazonaws.com/xxxbucket/testfile
[INF] curl.cpp:prepare_url(4194): URL changed is http://xxxbucket.s3-us-west-1.amazonaws.com/testfile
[INF] curl.cpp:insertV4Headers(2267): computing signature [GET] [/testfile] [] []
[INF] curl.cpp:url_to_host(100): url is http://s3-us-west-1.amazonaws.com
[INF] curl.cpp:GetObjectRequest(2880): downloading... [path=/testfile][fd=9]
[DBG] curl.cpp:RequestPerform(1923): connecting to URL http://xxxbucket.s3-us-west-1.amazonaws.com/testfile
< HTTP/1.1 403 Forbidden
< x-amz-request-id: 8D514A317B200E19
< x-amz-id-2: xxxxxxxx+yFinPbA3Fvao=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Sat, 06 May 2017 03:17:30 GMT
< Server: AmazonS3
< Via: 1.1 fmdmzpr03
< Cache-Control: proxy-revalidate
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< Age: 0
<
[ERR] curl.cpp:RequestPerform(1974): ### CURLE_WRITE_ERROR
@ggtakec commented on GitHub (May 6, 2017):
@umamaheswararao
In the log, the Get Object result is a 403 error.
Get Object's 403 error has the following possibilities:
Please check whether ListBucket operation is permitted.
@umamaheswararao commented on GitHub (May 7, 2017):
@ggtakec , Thank you so much for the reply.
I set the Bucket policy as below.
{
"Version": "2012-10-17",
"Id": "Policy1494101249587",
"Statement": [
{
"Sid": "Stmt1494101239585",
"Effect": "Allow",
"Principal": "",
"Action": "s3:",
"Resource": "arn:aws:s3:::xxxxxbucket1"
}
]
}
Since it is * , it should have covered all of that actions I think right. I am not sure what I am missing.
Also please note, I am able to create and write to files. only tail or cat is failing.
Working commands:
mkdir, touch, ls
Non working commands:
tail , cat
@ggtakec commented on GitHub (May 7, 2017):
@umamaheswararao
May I ask about options (command line) when starting up s3fs?
If it is possible to debug with dbglevel = dbg, curldbg, could you find a suspicious log?
@sqlbot commented on GitHub (May 7, 2017):
In the bucket policy, this isn't correct/complete:
The above only allows access to the bucket, not the objects in the bucket. For both, I believe it should look like this:
It's not clear to me why you are able to create files with the current policy, though there are multiple factors contributing to S3 permissions, including bucket ACLs and IAM user policies, so you might be picking up that permission via a different route.
Note also that this is very dangerous, for two reasons:
The most obvious danger is that this makes your bucket completely and totally public. Anyone can do anything to your bucket.
But this also could be related to your problem, since when a bucket is public like this, it is possible in some cases to create objects in the bucket that you don't actually own. These objects, under the right conditions, cannot be read by the bucket owner and this is a condition that cannot be solved. The only thing you can do in this case is to delete these objects.
It's probably better to use IAM user policies rather than bucket policies. There is an AmazonS3FullAccess managed policy that you can directly attach to an IAM user, for simplicity of troubleshooting your initial deployment (assuming this is what you are doing). You shouldn't use this in production since it gives that IAM user full access to all privileges on all buckets, but it is an easier way to avoid getting stuck on permissions, and is still far more secure than making a bucket public.
@umamaheswararao commented on GitHub (May 19, 2017):
@ggtakec , thank you for the reply...Sorry for my late response.
I tried with debug logs. below are few error logs:
[ERR] curl.cpp:RequestPerform(1974): ### CURLE_WRITE_ERROR
[INF] curl.cpp:RequestPerform(2082): ### retrying...
[INF] curl.cpp:RemakeHandle(1737): Retry request. [type=4][url=http://xxxxbucket.s3-us-west-1.amazonaws.com/myfile][path=/myfile]
[ERR] curl.cpp:RequestPerform(2089): ### giving up
[DBG] curl.cpp:ReturnHandler(306): Return handler to pool: 31
[ERR] fdcache.cpp:Read(1600): could not download. start(0), size(4096), errno(-5)
[WAN] s3fs.cpp:s3fs_read(2141): failed to read file(/myfile). result=-5
[DBG] fdcache.cpp:Close(2123): [ent->file=/myfile][ent->fd=5]
[DBG] fdcache.cpp:Close(697): [path=/myfile][fd=5][refcnt=1]
unique: 115, error: -5 (Input/output error), outsize: 16
unique: 116, opcode: FLUSH (25), nodeid: 7, insize: 64
flush[5]
[INF] s3fs.cpp:s3fs_flush(2185): [path=/myfile][fd=5]
[DBG] s3fs.cpp:check_parent_object_access(669): [path=/myfile]
[DBG] s3fs.cpp:check_object_access(563): [path=/]
[DBG] s3fs.cpp:get_object_attribute(416): [path=/]
[DBG] s3fs.cpp:check_object_access(563): [path=/myfile]
[DBG] s3fs.cpp:get_object_attribute(416): [path=/myfile]
I was trying to do tail myfile from the mounted directory.Write mayfile was success.
@umamaheswararao commented on GitHub (May 19, 2017):
@sqlbot Thank you so much for the advices. Actually this is just testing environment and just wanted to things work first, then I will tighten things about security configurations. Please look at my above error log if it gives some clue. Thank you
I tried below too...but no luck
"Resource": [ "arn:aws:s3:::xxxxxbucket1",
"arn:aws:s3:::xxxxxbucket1/*" ]
@umamaheswararao commented on GitHub (May 19, 2017):
I googled a bit and found https://github.com/s3fs-fuse/s3fs-fuse/issues/462
after passing -o url=https://s3.amazonaws.com, it worked
Thanks all for your time. Could you guys relate what was the issue? do you guys think this need to update in project readme or FAQ at least?