mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-24 21:06:02 +03:00
[GH-ISSUE #2570] Memory spike from 80% to 100% with s3fs processes #1237
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#1237
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Bhagyashreek8 on GitHub (Oct 24, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2570
Additional Information
Version of s3fs being used (
s3fs --version)Tried on V1.94 and V1.93
Version of fuse being used (
pkg-config --modversion fuse,rpm -qi fuseordpkg -s fuse)Fuse not installed
Kernel information (
uname -r)Linux kube-c0qiuvml0g8hqam57mvg-qcoscluster-qctestd-0000dd83 4.18.0-553.16.1.el8_10.x86_64 https://github.ibm.com/alchemy-containers/customer-tickets/issues/1 SMP Thu Aug 1 04:16:12 EDT 2024 x86_64 x86_64 x86_64 GNU/Linu
GNU/Linux Distribution, if applicable (
cat /etc/os-release)How to run s3fs, if applicable
[x] command line
[] /etc/fstab
s3fs syslog messages (
grep s3fs /var/log/syslog,journalctl | grep s3fs, ors3fs outputs)Details about issue
For one our customers on the kubernetes cluster, s3fs-fuse is spiking the Memory usage to 100% over a period of 2 days time. when s3fs process mounts a bucket from inside a cluster worker node, the worker node memory spikes to 100% by the s3fs process.
@Bhagyashreek8 commented on GitHub (Oct 24, 2024):
strace output for s3fs process
strace-output.txt
@ggtakec commented on GitHub (Oct 24, 2024):
Does your environment(including your customers) have a container running on a kubernetes worker node host and a corresponding s3fs process running on that worker node host?
(Or is the s3fs process running inside the container?)
I'm sorry I looked at the strace log, but it didn't help us identify the cause.
You've specified the
dbglevel=warnoption, but is there anything in the output that can solve the problem?(I think if possible, a
dbglevel=infolog would help us proceed with the analysis.)It might be difficult, but is it possible to use s3fs built with the master code(which has some bugs fixed) ?
@Bhagyashreek8 commented on GitHub (Oct 24, 2024):
Thank you for looking into it @ggtakec
s3fs process runs on the worker node host. A workload consuming the s3fs mount will run as a container (pod/ deployment) on kubernetes worker node. But the s3fs-fuse binary will be sitting on the worker node and s3fs process runs from that worker node host.
I will try to get more logs with
dbglevel=infofrom the customer cluster.s3fs built with master code would be difficult option as of now as the issue is happening only with the customer production cluster.
And we could not reproduce the issue on our test cluster.
@Bhagyashreek8 commented on GitHub (Nov 5, 2024):
Hi @ggtakec Here are the logs with debug level info. could you please take a look.
10-242-0-4_ibmc-s3fs.zip
@suresh9494 commented on GitHub (Nov 6, 2024):
Hello Team, We are touching base with you to see if there is any update on this . Thanks
@sivamtkr1102 commented on GitHub (Nov 10, 2024):
Hello @ggtakec , Request your help to prioritize this case. Thanks.
@ggtakec commented on GitHub (Nov 10, 2024):
Thanks for the ping.
Regarding this matter, I understand that you start s3fs on the worker node and use the mount point Volume from the container. Is that correct?
Currently, the information you have given me is unable to identify the cause of memory depletion (whether it is a bug or low memory allocation).
However, I was able to find a few interesting articles.
If possible, would it be possible to try removing the
kernel_cacheoption when starting s3fs?I have seen articles saying that there may be problems if this option is specified in k8s's pod.
@Bhagyashreek8 commented on GitHub (Nov 11, 2024):
@ggtakec Thanks for the response.
Hi @suresh9494 Could you please ask the customer to try as suggested above.
The
coldandvaultbased storage classes provided by s3fs plugin by default come withkernel-cacheset to false. Could you please try using one of these storage classes and redo the above exercise with debug level set to infodbglevel=infoand see if that works.@Bhagyashreek8 commented on GitHub (Nov 29, 2024):
Hi @ggtakec
However, they have mount s3fs to a different cos bucket (which has minimal - around 2 GB of data) and the memory spike has been reduced drastically. It is behaving normal.
The bucket they had mount earlier had > 100GB data with multiple files and folders.
Would there be any issue from s3fs side mounting volume to a cos bucket with huge data and accessing from regularly for read/write?
@suresh9494 - Could you add the info on how frequently customer would hit read/write requests to cos bucket?
@tanguofu commented on GitHub (Dec 15, 2024):
can you try with re complie s3fs with jemalloc ? Earlier versions of glibc might not timely release memory back to the operating system
@Bhagyashreek8 commented on GitHub (Dec 23, 2024):
@tanguofu Thank you for the response. Could you confirm the steps followed to build s3fs with jemalloc
Is there any other configuration setting required to build s3fs with jemalloc ?
cc @suresh9494
@tanguofu commented on GitHub (Dec 26, 2024):
@Bhagyashreek8
yes, this is dockerfile in my fork https://github.com/tanguofu/s3fs-fuse/blob/master/docker/cosfs-dockerfile#L24
@Bhagyashreek8 commented on GitHub (Dec 26, 2024):
Hi @tanguofu I tried to build s3fs-fuse with jemalloc, but I have an issue with installing jemalloc on my base image.
To give some background, we build s3fs-fuse on ubi-minimal as base image. And ubi-minimal does not have jemalloc in its microdnf repositories.
One way I can have jemalloc installed is getting the rpm and copying it on to the base image. It requires some effort. So I have couple of questions before trying that -
@Bhagyashreek8 commented on GitHub (Dec 26, 2024):
Hi @ggtakec Could you please check the above comment on priority - https://github.com/s3fs-fuse/s3fs-fuse/issues/2570#issuecomment-2507079751
@Bhagyashreek8 commented on GitHub (Jan 9, 2025):
Hi @ggtakec any update on this issue?
@ggtakec commented on GitHub (Jan 18, 2025):
@Bhagyashreek8 Sorry for my late reply.
I still don't understand the cause of this issue.
And I don't think that s3fs memory usage is affected by the bucket (the number of files in it, the file size), so I don't understand the cause of this comment.
(If you specify the
use_cacheoption, the cache file will increase, but since you didn't specify this, it doesn't matter.)Is it possible to perform a test without specifying the
kernel_cacheoption?(I am concerned about the impact of this option.)
Also, could you try it without specifying the
max_background=1000option?These options affect the operation of FUSE itself, not the operation of s3fs, so I would like to eliminate them first and see the results.