mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 13:26:00 +03:00
[GH-ISSUE #1245] Increased “Transport endpoint is not connected” rate with 1.86 #665
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#665
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @acdha on GitHub (Feb 14, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1245
Additional Information
I've noticed what appears to be an increased rate of “transport endpoint is not connected” errors with 1.86, although that could be a coincidence due to load — this has been a long-running problem which I was hoping 1.86 would improve.
Version of s3fs being used (s3fs --version)
Amazon Simple Storage Service File System V1.86 (commit:unknown) with OpenSSL
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
2.9.7-1ubuntu1
Kernel information (uname -r)
4.15.0-1058-aws
GNU/Linux Distribution, if applicable (cat /etc/os-release)
/etc/fstab entry, if applicable
s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)
Details about issue
I have a cron task which runs every minute to remount dead s3fs mountpoints after s3fs crashes or hangs so the logs are a bit noisy from those new processes starting.
@gaul commented on GitHub (Feb 18, 2020):
This is not expected behavior. Can you run again with
-f -dand share the logs?@acdha commented on GitHub (Mar 2, 2020):
This looks like it may be related to #1191 — it's happening many times per day and we definitely use
ensure_diskfree.output from -f -d
@IanTangney commented on GitHub (Mar 10, 2020):
Same issue after upgrading to 1.86, resolved after downgrading to 1.85
This is on a CentOS7 system, with the version of s3fs off of EPEL.
I'm not using
ensure_diskfree. Here is the entry in /etc/fstabs3fs#[bucket_name] /path/to/mount fuse umask=007,uid=[username],gid=[groupname],allow_other,url=[s3_url],use_cache=/mnt/fuse-cache 0 0The cache filesystem disk utilization never exceeded 75% of its capacity.
I wasn't able to capture the debug logs, because my priority was on resolving the errors.
@woodcoder commented on GitHub (Mar 14, 2020):
Same issue, with some investigation it appears to be a cache .stat file getting corrupted:
I believe that what's happening is that the stat file is being overwritten with a 0 total size, but then not being truncated.
@dfslezak commented on GitHub (Mar 14, 2020):
I am having the same problem. While this is solved, which version do you recommend to avoid this problem? Thanks
@IanTangney commented on GitHub (Mar 15, 2020):
@dfslezak
I do not experience this issue anywhere near as frequently on 1.85.
@gaul commented on GitHub (Apr 8, 2020):
Could you test with
-o nomixupload? s3fs 1.86 introduced this optimization and it may be the cause of your symptoms.@acdha commented on GitHub (Apr 8, 2020):
I can test but I would expect to make no difference since the workload where I see frequent failures is read-only.
@acdha commented on GitHub (Apr 8, 2020):
Confirmed, the errors happen with
nomixuploadmount option as well. Deleting the stat caches seems to help but it still crashes pretty frequently under load.@Morc001 commented on GitHub (Apr 14, 2020):
Maybe this is just repeating the same, maybe not, but what I noticed is that it always hangs after "file locked" when it's not followed by a "file unlocked". I see no error message in syslog. I don't use ensure_diskfree as s3fs has a separate cache partition. I mount with these mount options:
nodev,nosuid,allow_other,_netdev,use_sse,endpoint=eu-central-1,use_cache=/var/lib/s3fs/cache,enable_noobj_cache,max_stat_cache_size=500000
@woodcoder commented on GitHub (Apr 14, 2020):
I put together a quick fix/workaround for the corrupted stat files I discovered:
https://github.com/s3fs-fuse/s3fs-fuse/issues/1245#issuecomment-599047227
and this has been running with no issue for me now for over a month.
As I say, I believe the first character of the stat files are being overwritten with a 0 character, but then not truncated so the size on a subsequent read is corrupted. This causes the
terminate called after throwing an instance of 'std::invalid_argument'exception ins3fs_strtoofftshown in @acdha's logging.(I wondered whether this maybe happens when a separate cleanup script deletes a file from the cache?)
The workaround patch simply looks for sizes that start with 0 but still have more characters in the buffer and doesn't try to load from disk. This seems to be working fine.
@ggtakec commented on GitHub (Apr 14, 2020):
@woodcoder As you say, the problem seems to have been the failure to truncate the file (stat) that shows the status of the cache file.
I'll upload the patch code immediately and, if the test passes, will merge it into master.
But if you are using s3fs with packages, I don't think this problem can be solved immediately.
Thanks for your kindness.
@Morc001 commented on GitHub (Apr 14, 2020):
@woodcoder I applied your patch to commit
feafb44but it did not resolve the problem entirely. Although the problem appears less frequently now, but still happens. The commit from today only adds a test, so should not matter.@ggtakec commented on GitHub (Apr 14, 2020):
@Morc001 If you can, please try the code of #1271(not yet merged now).
Thanks in advance for your assistance.
@Morc001 commented on GitHub (Apr 16, 2020):
@ggtakec Thanks, that seems to have worked! :) Thanks all for your input!
@ggtakec commented on GitHub (Apr 18, 2020):
@Morc001 Thanks for try to use it.
@gaul commented on GitHub (Apr 22, 2020):
Fixed by #1271.