mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-26 05:45:57 +03:00
[GH-ISSUE #1215] V1.85 Transport Endpoint & V1.82 - CPU usage #646
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#646
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ALambert-TSV on GitHub (Dec 10, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1215
Version of s3fs being used (s3fs --version)
1.82 / 1.85
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
2.9.7
Kernel information (uname -r)
4.15.0-1056-aws
GNU/Linux Distribution, if applicable (cat /etc/os-release)
Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-1056-aws x86_64)
s3fs command line used, if applicable
-o use_cache=/AWSCACHE -o allow_other -o nonempty -o uid=1002 -o mp_umask=002 -o stat_cache_expire=3600 -o max_stat_cache_size=10000 -o url=https://s3.amazonaws.com -o dbglevel=warn
/etc/fstab entry, if applicable
UUID=ef6ba050-6cdc-416a-9380-c14304d0d206 / xfs defaults 0 0
fuse.s3fs _netdev,nonempty,allow_other,use_cache=/AWSCACHE,uid=1002,mp_umask=002,stat_cache_expire=3600,max_stat_cache_size=10000,url=https://s3.amazonaws.com, dbglevel=warn 0 0
s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)
V1.85:
curl.cpp:RequestPerform(2397): ###curlCode: 77 msg: Problem with the SSL CA cert (path? access rights?)
s3fs.cpp:list_bucket(2597): ListBucketRequest returns with error.
s3fs.cpp:directory_empty(1157): list_bucket returns error.
Details about issue
Have recently updated our s3fs install and
when running in version 1.85 of s3fs-fuse via git clone method, we get a Transport Endpoint error continually with a curlCode 77 issue. This results in the mount being unusable
When running V1.82 installed using sudo apt install s3fs, the s3fs process is using between 60% and 170% of the CPU on the server
The S3 bucket is being used to facilitate a public webpage so its fairly important that our organisation can sort this out.
@ALambert-TSV commented on GitHub (Dec 10, 2019):
Peaked at 190% CPU and only up for 74 minutes
@gaul commented on GitHub (Dec 10, 2019):
curlCode 77 corresponds to
CURLE_SSL_CACERT_BADFILE. This suggests some SSL misconfiguration. If you enable debug logs via-d -o curldbgwhat do you find?@ALambert-TSV commented on GitHub (Dec 11, 2019):
I dropped mounts, uninstalled s3fs completely and re-installed.. the curlCode 77 is now no longer happening.. However the mount is still dropping with Transport Endpoint every few minutes.
With debug mode on viewing the syslogs, the mount drop is occurring at this point:
Dec 11 13:28:46 (Servername} s3fs[3546]: fdcache.cpp:Open(254): file locked((Filename) - /AWSCACHE/.(S3 Bucket).stat/(FolderPath))
Dec 11 13:30:01 (Servername} amazon-ssm-agent.amazon-ssm-agent[850]: 2019-12-11 13:30:01 INFO [HealthCheck] HealthCheck reporting agent health.
Dec 11 13:30:01 (Servername} CRON[3718]: (ubuntu) CMD (/home/bvcameraftp/checkmount.sh >> /home/bvcameraftp/checkmount.log)
Dec 11 13:30:01 (Servername} CRON[3719]: (root) CMD (/opt/McAfee/agent/scripts/ma checkhealth >/dev/null 2>/dev/null)
Dec 11 13:30:01 (Servername} systemd[1]: Configuration file /etc/systemd/system/mcafee.ma.service is marked executable. Please remove executable permission bits. Proceeding anyway.
Dec 11 13:30:02 (Servername} CRON[3717]: (CRON) info (No MTA installed, discarding output)
Dec 11 13:30:02 (Servername} systemd[1]: Started Session 30 of user mfe.
Dec 11 13:30:03 (Servername} systemd[1]: Created slice User Slice of boatingvic.
Dec 11 13:30:03 (Servername} systemd[1]: Starting User Manager for UID 1001...
Dec 11 13:30:03 (Servername} systemd[1]: Started Session 32 of user boatingvic.
Dec 11 13:30:03 (Servername} systemd[1]: Started Session 33 of user boatingvic.
Dec 11 13:30:03 (Servername} systemd[3759]: Reached target Timers.
Dec 11 13:30:03 (Servername} systemd[3759]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Dec 11 13:30:03 (Servername} systemd[3759]: Listening on GnuPG cryptographic agent and passphrase cache.
Dec 11 13:30:03 (Servername} systemd[3759]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Dec 11 13:30:03 (Servername} systemd[3759]: Listening on REST API socket for snapd user session agent.
Dec 11 13:30:03 (Servername} systemd[3759]: Listening on GnuPG network certificate management daemon.
Dec 11 13:30:03 (Servername} systemd[3759]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Dec 11 13:30:03 (Servername} systemd[3759]: Reached target Sockets.
Dec 11 13:30:03 (Servername} systemd[3759]: Reached target Paths.
Dec 11 13:30:03 (Servername} systemd[3759]: Reached target Basic System.
Dec 11 13:30:03 (Servername} systemd[3759]: Reached target Default.
Dec 11 13:30:03 (Servername} systemd[3759]: Startup finished in 64ms.
Dec 11 13:30:03 (Servername} systemd[1]: Started User Manager for UID 1001.
Dec 11 13:30:05 (Servername} systemd[1]: Started Session 35 of user boatingvic.
Dec 11 13:30:05 (Servername} systemd[1]: Started Session 36 of user boatingvic.
Dec 11 13:30:05 (Servername} systemd[1]: Started Session 37 of user boatingvic.
Dec 11 13:30:05 (Servername} systemd[1]: Started Session 38 of user boatingvic.
Dec 11 13:30:07 (Servername} systemd[1]: Started Session 39 of user boatingvic.
Dec 11 13:30:07 (Servername} systemd[1]: Stopping User Manager for UID 1001...
Dec 11 13:30:07 (Servername} systemd[3759]: Stopped target Default.
Dec 11 13:30:07 (Servername} systemd[3759]: Stopped target Basic System.
Dec 11 13:30:07 (Servername} systemd[3759]: Stopped target Sockets.
Dec 11 13:30:07 (Servername} systemd[3759]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Dec 11 13:30:07 (Servername} systemd[3759]: Closed REST API socket for snapd user session agent.
Dec 11 13:30:07 (Servername} systemd[3759]: Stopped target Paths.
Dec 11 13:30:07 (Servername} systemd[3759]: Closed GnuPG network certificate management daemon.
Dec 11 13:30:07 (Servername} systemd[3759]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Dec 11 13:30:07 (Servername} systemd[3759]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Dec 11 13:30:07 (Servername} systemd[3759]: Closed GnuPG cryptographic agent and passphrase cache.
Dec 11 13:30:07 (Servername} systemd[3759]: Reached target Shutdown.
Dec 11 13:30:07 (Servername} systemd[3759]: Starting Exit the Session...
Dec 11 13:30:07 (Servername} systemd[3759]: Stopped target Timers.
Dec 11 13:30:07 (Servername} systemd[3759]: Received SIGRTMIN+24 from PID 4371 (kill).
Dec 11 13:30:07 (Servername} systemd[1]: Started Session 40 of user boatingvic.
Dec 11 13:30:07 (Servername} systemd[1]: Starting User Manager for UID 1001...
Dec 11 13:30:07 (Servername} systemd[4372]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Dec 11 13:30:07 (Servername} systemd[4372]: Listening on GnuPG cryptographic agent and passphrase cache.
Dec 11 13:30:07 (Servername} systemd[4372]: Reached target Paths.
Dec 11 13:30:07 (Servername} systemd[4372]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Dec 11 13:30:07 (Servername} systemd[4372]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Dec 11 13:30:07 (Servername} systemd[4372]: Listening on GnuPG network certificate management daemon.
Dec 11 13:30:07 (Servername} systemd[4372]: Listening on REST API socket for snapd user session agent.
Dec 11 13:30:07 (Servername} systemd[4372]: Reached target Sockets.
Dec 11 13:30:07 (Servername} systemd[4372]: Reached target Timers.
Dec 11 13:30:07 (Servername} systemd[4372]: Reached target Basic System.
Dec 11 13:30:07 (Servername} systemd[4372]: Reached target Default.
Dec 11 13:30:07 (Servername} systemd[4372]: Startup finished in 16ms.
Dec 11 13:30:07 (Servername} systemd[1]: Started User Manager for UID 1001.
Dec 11 13:30:08 (Servername} systemd[1]: Stopping User Manager for UID 1001...
Dec 11 13:30:08 (Servername} systemd[4372]: Stopped target Default.
Dec 11 13:30:08 (Servername} systemd[4372]: Stopped target Basic System.
Dec 11 13:30:08 (Servername} systemd[4372]: Stopped target Sockets.
Dec 11 13:30:08 (Servername} systemd[4372]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Dec 11 13:30:08 (Servername} systemd[4372]: Closed GnuPG network certificate management daemon.
Dec 11 13:30:08 (Servername} systemd[4372]: Closed REST API socket for snapd user session agent.
Dec 11 13:30:08 (Servername} systemd[4372]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Dec 11 13:30:08 (Servername} systemd[4372]: Closed GnuPG cryptographic agent and passphrase cache.
Dec 11 13:30:08 (Servername} systemd[4372]: Stopped target Paths.
Dec 11 13:30:08 (Servername} systemd[4372]: Stopped target Timers.
Dec 11 13:30:08 (Servername} systemd[4372]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Dec 11 13:30:08 (Servername} systemd[4372]: Reached target Shutdown.
Dec 11 13:30:08 (Servername} systemd[4372]: Starting Exit the Session...
Dec 11 13:30:08 (Servername} systemd[4372]: Received SIGRTMIN+24 from PID 4468 (kill).
Dec 11 13:30:08 (Servername} systemd[1]: Stopped User Manager for UID 1001.
Dec 11 13:30:08 (Servername} systemd[1]: Removed slice User Slice of (user).
^Z
[8]+ Stopped tail -f /var/log/syslog
ubuntu@(Servername):~$ df -Th
df: (Mountpoint): Transport endpoint is not connected
@ALambert-TSV commented on GitHub (Dec 15, 2019):
further testing has confirmed that the amazon-ssm-agent.amazon-ssm-agent is not the cause as this issue still occurs when ssm healthcheck is turned off
@gaul commented on GitHub (Feb 3, 2020):
Please run s3fs with debug logs
-f -dand share the results.@gaul commented on GitHub (Jun 4, 2020):
Alternatively try to connect
gdbto the process and share the backtrace. Closing due to inactivity.