[GH-ISSUE #936] Consider removing malloc_trim calls #529

New issue

Closed

opened 2026-03-04 01:46:25 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:46:25 +03:00

Owner

Originally created by @gaul on GitHub (Jan 29, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/936

s3fs issues malloc_trim after every FUSE operation and XML parsing. This has a lot of overhead with many threads they all call malloc_trim and each examines the entire heap, an O(n**2) operation. 42b74c9d2e introduced these calls. @ggtakec could you share more background on this since the change was intermingled with others? References #935.

Originally created by @gaul on GitHub (Jan 29, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/936 s3fs issues `malloc_trim` after every FUSE operation and XML parsing. This has a lot of overhead with many threads they all call `malloc_trim` and each examines the entire heap, an O(n**2) operation. 42b74c9d2ed67da4cf1f6586d179b1d3c34fcb10 introduced these calls. @ggtakec could you share more background on this since the change was intermingled with others? References #935.

kerem closed this issue

2026-03-04 01:46:25 +03:00

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 29, 2019):

@gaul
There are several reasons why we were using malloc_trim.
It's a macro built into the old version.
The current s3fs almost no memory leak is detected.
However, I corrected and investigated the memory leak of the old version, and then incorporated this macro at that time.

There was also a memory leak as a defect, but there were reports by valgrind which can not be understand.
I incorporated libxml2/malloc_trim with reference to the document of http://xmlsoft.org/xmlmem.html.
And that unrecognizable report was settled.

In other words, by allocating/freeing a small memory area, memory fragmentation occurred and the process memory was increased.
I checked using malloc_trim, MMAP_THRESHOLD etc, and confirmed that what is written in this document occurred as a result of investigation.

I had anticipated that the increase in memory size would stop at a certain upper limit, but there were other memory leaks, it did not stop and was fatal.
This state was a problem because s3fs has been running for a long time as a daemon process.
From this background, we will use this macro to minimize the memory expansion of the daemon process.

BUT, the latest version is that the memory leak is no longer detected, and it seems that there is no effect of bloating due to fragmented memory by libxml2.
I think that calling malloc_trim directly should stop, if nothing is more costly than anything else.(originally it is called inside the free function)

I think that it is good to stop calling malloc_trim by #937 PR.
(I will write a comment on PR on how to fix it)

@ggtakec commented on GitHub (Jan 29, 2019): @gaul There are several reasons why we were using malloc_trim. It's a macro built into the old version. The current s3fs almost no memory leak is detected. However, I corrected and investigated the memory leak of the old version, and then incorporated this macro at that time. There was also a memory leak as a defect, but there were reports by valgrind which can not be understand. I incorporated libxml2/malloc_trim with reference to the document of http://xmlsoft.org/xmlmem.html. And that unrecognizable report was settled. In other words, by allocating/freeing a small memory area, memory fragmentation occurred and the process memory was increased. I checked using malloc_trim, MMAP_THRESHOLD etc, and confirmed that what is written in this document occurred as a result of investigation. I had anticipated that the increase in memory size would stop at a certain upper limit, but there were other memory leaks, it did not stop and was fatal. This state was a problem because s3fs has been running for a long time as a daemon process. From this background, we will use this macro to minimize the memory expansion of the daemon process. BUT, the latest version is that the memory leak is no longer detected, and it seems that there is no effect of bloating due to fragmented memory by libxml2. I think that calling malloc_trim directly should stop, if nothing is more costly than anything else.(originally it is called inside the free function) I think that it is good to stop calling malloc_trim by #937 PR. (I will write a comment on PR on how to fix it)