mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 21:35:58 +03:00
[GH-ISSUE #1883] s3fs slowness when using sftp. Looking for non-existent files/directories #959
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#959
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @arukaen on GitHub (Feb 3, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1883
Version of s3fs being used (s3fs --version)
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
Kernel information (uname -r)
5.11.0-1028-aws
GNU/Linux Distribution, if applicable (cat /etc/os-release)
/etc/fstab entry, if applicable
s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)
Details about issue
I am running an SFTP server that is chrooting users to
/data/user./datais the mount point for s3fs andmy-bucket. My instance and the bucket are both in the us-east-1 region of AWS. When I attempt to connect to the sftp (sftp user@endpoint), after entering my password I have about a 12 second delay until I am able to see my directory (which currently has 0 files). This happens on any SFTP client. When I look at the logs, I notice that there are calls looking for directories that don't exist (nor have I specified them anywhere). For example:.cache, .ssh, proc, etc. When I created the users, I make sure to use a custom/etc/skel2which just creates a empty directory for the users home. Why are these files being looked for?@ggtakec commented on GitHub (Feb 6, 2022):
If you use a bucket mounted with s3fs that doesn't have objects uploaded from anything other than s3fs, you can use the
notsup_compat_diroption.By granting this option, unnecessary communication can be eliminated.
However, specifying this option can result in unrecognized directories(middle paths to file objects) in s3fs.
Alternatively, you can reduce the traffic by specifying the
enable_noobj_cacheoption.See the description of the
notsup_compat_dirandenable_noobj_cacheoptions for more information.@arukaen commented on GitHub (Feb 7, 2022):
The bucket in question is currently empty. The plan is for this bucket/instance to replace an existing SFTP server. If I copy (aws s3 sync) from the old server to said bucket and have
notsup_compat_diron, what kinda issues can I run into?@CarstenGrohmann commented on GitHub (Feb 19, 2022):
Quote from #1892 - this may answer your question:
@ggtakec commented on GitHub (Feb 20, 2022):
You may not be able to see the file containing the directory path(ex.
<Mount point>/<dir>/file).Since the above
<dir>will not be recognized(<dir>does not exist as an S3 object), you will probably not be able to see the files under that directory path.@arukaen commented on GitHub (Feb 23, 2022):
Thanks for the help!