mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 13:26:00 +03:00
[GH-ISSUE #2054] Memory Leak on s3fs fuse Ubuntu 20.04 #1040
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#1040
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @johnh530 on GitHub (Nov 7, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2054
Additional Information
We run an intensive processing of about 1MB files stores on s3 bucket which are about 3MB to (now) limited to 30MB in size.
We typically run 3-5 processes concurrently on this 4GB machine.
On a 4 GB machine with the s3fs just mounted the memory consumption is 0.3% after the run is completed and no other s3fs activity the memory consumption is 9.6%
Start batch
102063 root 0.3 s3fs gedmatch-dna-files-v2-full-dev /opt/gedmatchV1/databases/dna-files/V2_full_kits -o rw,endpoint=us-west-2,nonempty,allow_other,mp_umask=0002,umask=0002,uid=33,gid=33,dev,suid
End batch
PID USER %MEM COMMAND
102063 root 9.6 s3fs gedmatch-dna-files-v2-full-dev /opt/gedmatchV1/databases/dna-files/V2_full_kits -o rw,endpoint=us-west-2,nonempty,allow_other,mp_umask=0002,umask=0002,uid=33,gid=33,dev,suid
Unmounting and remounting the file system returns it back to 0.3% memory use.
Version of s3fs being used (s3fs --version)
Amazon Simple Storage Service File System V1.86 (commit:unknown) with GnuTLS(gcrypt)
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 113
Maintainer: Ubuntu Developers ubuntu-devel-discuss@lists.ubuntu.com
Architecture: amd64
Version: 2.9.9-3
Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
/etc/fuse.conf 298587592c8444196833f317def414f2
Kernel information (uname -r)
5.15.0-1022-aws
GNU/Linux Distribution, if applicable (cat /etc/os-release)
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
s3fs command line used, if applicable
/etc/fstab entry, if applicable
gedmatch-dna-files-v2-full-dev /opt/gedmatchV1/databases/dna-files/V2_full_kits fuse.s3fs _netdev,endpoint=us-west-2,nonempty,allow_other,mp_umask=0002,umask=0002,uid=33,gid=33 0 0
s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)
if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages
Details about issue
Curl info
ii libcurl3-gnutls:amd64 7.68.0-1ubuntu2.14 amd64 easy-to-use client-side URL transfer library (GnuTLS flavour)
ii libcurl4:amd64 7.68.0-1ubuntu2.14 amd64 easy-to-use client-side URL transfer library (OpenSSL flavour)
open ssl info
ii libgnutls-openssl27:amd64 3.6.13-2ubuntu1.7 amd64 GNU TLS library - OpenSSL wrapper
ii libxmlsec1-openssl:amd64 1.2.28-2 amd64 Openssl engine for the XML security library
ii openssl 1.1.1f-1ubuntu2.16 amd64 Secure Sockets Layer toolkit - cryptographic utility
I see other issues related to memory leaks indicating possible issues in curl/ssl - so this was included.
Is it unreasonable to expect that s3fs should return to consuming only 0.3% after a batch run and there is no activity on s3fs?
johnh...
@ggtakec commented on GitHub (Nov 10, 2022):
@johnh530
You seem to be using version 1.86, but by v1.91 some memory leak fixes have been made.
If possible, could you please try with v1.91?
@johnh530 commented on GitHub (Nov 11, 2022):
Have built but not tried v1.91 yet - In looking over the release notes I saw one item which looked like a memory leak related to calls which I don't think I was using but hopefully will be able to take a look this weekend.
johnh...
From: Takeshi Nakatani @.>
Sent: Thursday, November 10, 2022 5:53 AM
To: s3fs-fuse/s3fs-fuse @.>
Cc: John Hayward @.>; Mention @.>
Subject: Re: [s3fs-fuse/s3fs-fuse] Memory Leak on s3fs fuse Ubuntu 20.04 (Issue #2054)
@johnh530https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjohnh530&data=05%7C01%7CJohn.Hayward%40wheaton.edu%7C51068fbbff6c4cbf868708dac322f994%7Cb7098c8ac6b24e8bba4c872cf5f00a20%7C0%7C0%7C638036852238641522%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=k%2FxzOBGEjhCVDIX9vOjqHFT%2B131a9%2Fxm1N3E%2FUjDWbU%3D&reserved=0
You seem to be using version 1.86, but by v1.91 some memory leak fixes have been made.
If possible, could you please try with v1.91?
—
Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fs3fs-fuse%2Fs3fs-fuse%2Fissues%2F2054%23issuecomment-1310313830&data=05%7C01%7CJohn.Hayward%40wheaton.edu%7C51068fbbff6c4cbf868708dac322f994%7Cb7098c8ac6b24e8bba4c872cf5f00a20%7C0%7C0%7C638036852238641522%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9vKxQmrscH401umJDrV8gCN9m8J9d0979LJ%2Bo02ysos%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAA7LDX7FP6C5TIXWFBBLPV3WHT46JANCNFSM6AAAAAARZQULGM&data=05%7C01%7CJohn.Hayward%40wheaton.edu%7C51068fbbff6c4cbf868708dac322f994%7Cb7098c8ac6b24e8bba4c872cf5f00a20%7C0%7C0%7C638036852238641522%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JKUHLeapcNzyxC9wlhVqFQwRS50B7p0fjBItDysMBC4%3D&reserved=0.
You are receiving this because you were mentioned.Message ID: @.***>
@johnh530 commented on GitHub (Nov 15, 2022):
Have installed v1.91 and memory leak still seems to be there - running a smaller case here is what s3fs top looks like:
At start
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
259851 root 20 0 229244 9180 6060 S 0.0 0.2 0:00.00 s3fs
After run
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
259851 root 20 0 1162620 64132 6348 S 0.0 1.7 18:45.55 s3fs
After the run the memory used is 1.7% used with no activity on s3fs at start 0.2% was used.
So running the v1.91 still seems to be leaking memory.
In looking at the change log the most current mention of memory leaks being fixed are from 1.85:
...
#900 - Plug memory leak
#901 - Plug memory leaks
...
johnh...
@johnh530 commented on GitHub (Nov 15, 2022):
Did another run but this time rather than 5 concurrent processes running there was only one process running - Still shows a memory leak but not quite as bad. Started with 0.2% memory use and after completion at 0.9% memory use.
Start
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
265688 root 20 0 229244 9128 6008 S 0.0 0.2 0:00.00 s3fs
Completed
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
265688 root 20 0 585636 32092 6168 S 0.0 0.9 18:09.20 s3fs
johnh...
@ggtakec commented on GitHub (Mar 26, 2023):
@johnh530 I'm sorry for my late reply.
I tried run
valgrindon the s3fs master code and test scripts, but it didn't detect any memory leaks.(There is only one
CRYPTO_mallocleft, but I think this is a problem in libcrypto.)If you can, please try to run current master branch code.
Thanks in advance for your assistance.