mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 13:26:00 +03:00
[GH-ISSUE #344] large files fail to rsync (465M) #178
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#178
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @AlmirKadric on GitHub (Jan 29, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/344
When I rsync up large files (465M), the upload fails with the following rsync error:
I already bumped up the retry count to 10, but this didn't help
s3fs ${BUCKET} ${MNTDIR}/${BUCKET} -oretries=9;If there any other options I should try let me know and I'll give them a go.
OS: MacOSX Mavericks 10.10
s3fs-fuse: 1.79 (brew installed)
@postmaxin commented on GitHub (Jan 29, 2016):
I had the same issue, I ended up going with "-o parallel_count=2 -o retries=100" on the s3fs side, and "--timeout=600" on the rsync side.
I was also trying to upload very large files (25+GiB), so I ended up using "split" to chunk them into 4000MiB pieces before uploading.
@AlmirKadric commented on GitHub (Jan 30, 2016):
Thanks for the advice @postmaxin
Will give it a go and post back the results
Also why reduce the
parallel_countto 2? (default is 5)Is this to speed up individual multipart uploads?
@AlmirKadric commented on GitHub (Jan 30, 2016):
Gave the above a try and no luck. It seems this is an issue inside s3fs
I also ran s3fs in foreground debug mode and it seems that it at some point calls
s3fs_destroy(3281): destroyWhat's worse is I came across another problem, when destroy is called it fails to release the curl library
s3fs_destroy(3285): Could not release curl library.causing it to not kill the process leaving around junk pids. These pids are non-responsive and need to be killed with the-9signal to successfully cleanup the mess.Seems that something is causing s3fs to destroy and thus the connection that rsync was using ends up closed failing the sync.
Anyone got any ideas as to why?
Also when I checked the logs for the large file, I have no errors for the multipart uploads. They seem to be going through just fine, but at some point s3fs is killing the mount causing the failure. My guess is the large file is exposing an internal issues inside s3fs due to the volume of data.
Alternatively this could be an issue inside
osxfuseas it seems that s3fs does not calls3fs_oper.destroyinternally.@ggtakec commented on GitHub (Feb 6, 2016):
@AlmirKadric Could you run s3fs with "dbglevel" option(which is as same as -d option, but can specify log level) as foreground?
I want to know why call s3fs_destroy function, thus dbglevel option probably tell us the reason.
If you can, please try to do it.
Thanks in advance for your help.
@AlmirKadric commented on GitHub (Feb 6, 2016):
@ggtakec when i tried to use dbglevel i kept getting an unknown option error
however the following works for me:
could you please clarify what is the exact command you would like me to run?
@ggtakec commented on GitHub (Feb 6, 2016):
@AlmirKadric
Please try to run s3fs following:
dbglevel option is specified after "-o" parameter,.
Regards,
@AlmirKadric commented on GitHub (Feb 6, 2016):
@ggtakec same result
@ggtakec commented on GitHub (Feb 6, 2016):
@AlmirKadric Which version of s3fs do you use?
Since I had not been renewed branch of macosx, I just merge.
Could you try using the latest codes?
Thanks in advance for your help.
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec I'm using version "1.79" and installed it over homebrew.
Homebrew is using the following url:
https://github.com/s3fs-fuse/s3fs-fuse/archive/v1.79.tar.gzwhen you say latest do you mean develop? or 1.79 release???
and it seems I'm using
osxfuseversion:2.8.2@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec i just tried it with the
-ocurldbgand it worked fine, but still no luck with-odbglevel=dbgcould this have anything to do with an older version of fuse?
From whatI can see, it just stops working and the mount completely hangs.
I end up having to
kill -9the process and then unmount.@ggtakec commented on GitHub (Feb 7, 2016):
Hi, @AlmirKadric
You can download latest codes on master branch from following url:
https://github.com/s3fs-fuse/s3fs-fuse/archive/master.zip
It is fixed some bugs, please try to use it.
And you set dbglevel as from err, warn, info, to dbg, probabely many log is put.
We need to know what something wrong is occurred.
Thanks in advance for your help.
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec i got an error when trying to install master
@ggtakec commented on GitHub (Feb 7, 2016):
Ah...
I merged new code asap.
@ggtakec commented on GitHub (Feb 7, 2016):
@AlmirKadric I updated master branch by #355
Probably you can build s3fs, please try it.
Regards,
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec seems its a no go
@AlmirKadric commented on GitHub (Feb 7, 2016):
check this line, it seems to be the bug
github.com/s3fs-fuse/s3fs-fuse@1c9d7a9ea9/src/cache.cpp (L50)I'm assuming this should say if not defined then define?
but there its written if defined, define it again....
@AlmirKadric commented on GitHub (Feb 7, 2016):
i changed that line and it seems to compile now
FROM
TO
@AlmirKadric commented on GitHub (Feb 7, 2016):
and it works, i can mount and see dbg logs now
let me get back to you with my findings
@ggtakec commented on GitHub (Feb 7, 2016):
@AlmirKadric Thanks, but my these line is not safe, so will fix soon.
@ggtakec commented on GitHub (Feb 7, 2016):
Fixed by #356, and updated master branch.
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec
@ggtakec commented on GitHub (Feb 7, 2016):
I'm sorry and again and again. Updated by #357
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec INSTALLED!!!
ill try it now
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec I just ran my test, and it seems to be working fine now... getting no errors.
Will now try the larger test rather than a single file and will let you know once its done
(may take a an hour or so)
@AlmirKadric commented on GitHub (Feb 7, 2016):
@ggtakec the above fixed my issue with single large files. However when trying to
rsyncfolders of files i get broken pipe errors as follows:will switch to debug to investigate further
@AlmirKadric commented on GitHub (Feb 8, 2016):
@ggtakec the logs don't show anything in particular
however rsync keeps throwing
Interrupted system calland soon after the whole operation failsWith the previous version of s3fs this wasn't the case, instead it would only fail with large files
but now it works with large files but fails after a few file operations...
@ggtakec commented on GitHub (Feb 8, 2016):
@AlmirKadric s3fs didn't put any logs, did it?
If you run s3fs background(not specify -f option), probably s3fs puts logs under /var/log.
Please check the files in /var/log(syslog etc.)
And you do not use --inplace option for rsync, you should be set this option.
Because rsync replaces files after rsync sends them without this option.
Last, we need to know why rsync got error(errno 11 = EAGAIN) and who sent it.
Regards,
@ggtakec commented on GitHub (Mar 30, 2019):
We kept this issue open for a long time.
Is this problem continuing?
We launch new version 1.85, which fixed some problem(bugs).
Please use the latest version.
@LIII-XXII commented on GitHub (Apr 3, 2019):
I did run into a similar issue with 1.84, yes. I will retry with the latest version.
@gaul commented on GitHub (Jul 9, 2019):
Please test with the latest 1.85 version and open a new issue if the symptoms persist.