mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 13:26:00 +03:00
[GH-ISSUE #1483] Memory leak - s3fs using over 1GB memory #780
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#780
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @francoisfreitag on GitHub (Nov 25, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1483
Thank you for the great product. It’s been very reliable and greatly simplifies the setup for our use case.
The issue was noticed because the OOM-killer selects the s3fs process when the machine runs out of memory. The machine only serves as an SFTP server, where uploaded files are saved to an S3FS. It should not run out of memory.
Clients upload several hundred thousand files nightly. Monitoring the machine every day, the s3fs process memory consumes more and more memory until it exhausts the machine available RAM (about 2 GB).
Additional Information
Version of s3fs
Amazon Simple Storage Service File System V1.87 (commit:unknown) with OpenSSL
Version of fuse
2.9.2-11.el7Kernel information (uname -r)
3.10.0-1160.2.2.el7.x86_64GNU/Linux Distribution, if applicable (cat /etc/os-release)
CentOS 7
s3fs command line used, if applicable
Details about issue
Possibly related issues #340, #725.
Steps to reproduce
I created a Vagrantfile to precisely describe the setup and help reproducing the issue. The issue can be reproduced without it by following the steps in the
config.vm.provisionsection.<AWS_ACCESS_KEY_ID>,<AWS_SECRET_ACCESS_KEY>,<BUCKET_NAME>)vagrant upsftp testsftp@<MACHINE_ADDRESS>, the password ispassword. Can use a more complete SFTP client for faster transfers.put -R dataMonitor the s3fs process as files are transferred (e.g.
ps -eo pid,comm,rss), notice that it keeps growing without bounds. At the end of theput -R data, the S3 process uses several hundred MB.Script to generate many files
Vagrantfile
@ma331 commented on GitHub (Mar 26, 2021):
I have the exact same problem, but with a twist.
The same bucket is mounted twice (with same binary and exact same options), and only one s3fs process memleaks. the other doesnt. the one that does, moves a lot of files within s3 via sftp.
on the other s3mount without the problem files are only uploaded.
the difference is one process has 67mb rss the other 4gb.
used version is v1.89 build from git on mar. 13
@gaul commented on GitHub (Apr 24, 2021):
Could you share which SSL library you use, e.g., OpenSSL, GnuTLS? Apparently the latter has a memory leak: curl/curl#5102
@francoisfreitag commented on GitHub (Apr 24, 2021):
I’m using OpenSSL.
Using the Vagrantfile above, the versions are:
After running a system upgrade in the VM, they get upgraded to:
I can double check the steps to reproduce are still valid if that helps?
@gaul commented on GitHub (Apr 25, 2021):
If you can reproduce these symptoms it would be great. Since we cannot reproduce this locally using some leak checking tool like Valgrind's memcheck or massif would help.
@ma331 commented on GitHub (Apr 26, 2021):
im using gnutls here. ill recompile with openssl and check if the memory usage changes.
libcurl-gnutls.so.4 => /usr/lib/x86_64-linux-gnu/libcurl-gnutls.so.4 (0x00007f2f657a4000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f2f6497a000)
@francoisfreitag commented on GitHub (Apr 28, 2021):
Here are the steps I took:
setsebool use_fusefs_home_dirs onsed -ri 's/^PasswordAuthentication no$/PasswordAuthentication yes/' /etc/ssh/sshd_configsystemctl reload sshdsudo -iMeasure RSS size with
ps -eo pid,comm,rss:107376.sftp vagrant@<VM_IP_ADDRESS>Measure RSS size with
ps -eo pid,comm,rss:144072.The memory keeps growing. It will keep growing until it fills the available RAM, but that filling is very slow.
Here’s the output from valgrind.
valgrind.txt.gz
@gaul commented on GitHub (Apr 29, 2021):
Thanks for testing this! Valgrind says that there is no leak from its perspective which helps narrow this down. s3fs must be holding onto some memory unintentionally. Could you run this again with
valgrind --tool=massifwhich should show the major memory consumers at different points in time. Performance should be somewhat better than with the default memcheck. I expect that one of the caches is growing without bound.@francoisfreitag commented on GitHub (Apr 29, 2021):
Thanks for getting back to me, good to know this is helpful!
I’ll run the experiment with massif and report after the 100k files are uploaded.
@francoisfreitag commented on GitHub (Apr 30, 2021):
Here’s the file generated by massif: massif.out.2053.gz
@gaul commented on GitHub (May 9, 2021):
This suggests that s3fs only allocated maximum 5 MB of memory? Can you share the output from
pmap $(pidof s3fs)? I wonder if there is some other leak like pthreads or something.@francoisfreitag commented on GitHub (May 14, 2021):
Reran the experiment, here’s the output from
pmap $(pidof s3fs): pmap.out.gzAt the measurement time, the RSS size of s3fs is 20,920 (approx 20 MB).
@francoisfreitag commented on GitHub (May 18, 2021):
I keep uploading files, the memory keeps growing slowly. The RSS size of s3fs is now 22,260. Here’s the pmap output: pmap.out.2.gz
@qianyi-sourse commented on GitHub (Sep 6, 2023):
Thank you for the great product. It’s been very reliable and greatly simplifies the setup for our use case.
I have met the similar problem. In the mount directory, I created three directories, and in each directory, I uploaded 100000 objects. When I open these directories, the memory will grow very quickly. Ultimately, it will exceed 1GB.
##s3fs command
s3fs test-list /nfs/test5 -o passwd_file=/root/.passwd-s3fs -o url=http://xxx -o use_path_request_style -o noxmlns -o dbglevel=error -o default_acl=public-read -o logfile=/var/log/s3fs.log -o allow_other -o multireq_max=500 -o nocopyapi -o use_cache=/es-SP0-0/nfs/ -o del_cache -o parallel_count=500 -o multipart_size=52@gaul commented on GitHub (Sep 8, 2023):
-o parallel_count=500is not a good idea -- this means s3fs will use 500 threads. Can you try a smaller value? Similarly-o multireq_max=500seems excessively large.@qianyi-sourse commented on GitHub (Sep 8, 2023):
I have tried
-o parallel_count=100,-o multireq_max=100,but it did not work. Memory usage has not decreased.@gaul commented on GitHub (Sep 8, 2023):
These are stil unusually big values -- why aren't you using the default? Each thread will use a few MB of memory.
@qianyi-sourse commented on GitHub (Sep 11, 2023):
@gaul Thanks for your reply. I also have tried
-o parallel_count=5,-o multireq_max=5,but it did not work. Afterwards, I also tried to update to the latest code, but it didn't work.@gaul commented on GitHub (Sep 12, 2023):
What does "didn't work" mean? Did you have the same memory leak?
@qianyi-sourse commented on GitHub (Sep 13, 2023):
Yes. The change of
parallel_count,-o multireq_maxdid not reduce the usage of memory. Open a dir that has 100,000 files ,the usage of memory will eventually reach around 400MB. Then open other dir, the usage of memory will continue to increase.@qianyi-sourse commented on GitHub (Sep 14, 2023):
I checked the content in the memory through GDB and found that most of the data stored in the memory is HTTP response data.
@morkeleb commented on GitHub (Nov 30, 2023):
were experiencing the same issue. When we have folders with many files the memory runs out.
@pianetarosso commented on GitHub (Jan 12, 2024):
Same error, we are loading 5 different buckets, and everything works for the "smaller" ones, when I try to list the files in the bigger with 54K files we get this error
Out of memory: Killed process 49060 (s3fs) total-vm:851644kB, anon-rss:187516kB, file-rss:5120kB, shmem-rss:0kB, UID:0 pgtables:572kB oom_score_adj:0