[PR #2681] [MERGED] Refactor to change StatCache class and add StatCacheNode classes #2705

New issue

Closed

opened 2026-03-04 02:06:53 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:06:53 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/2681
Author: @ggtakec
Created: 6/6/2025
Status: ✅ Merged
Merged: 7/24/2025
Merged by: @gaul

Base: master ← Head: stat/04_statnode

📝 Commits (1)

3383fd7 Refactor to change StatCache class and add StatCacheNode classes

📊 Changes

16 files changed (+3012 additions, -1394 deletions)

View changed files

📝 src/Makefile.am (+1 -0)
📝 src/cache.cpp (+210 -615)
📝 src/cache.h (+31 -103)
➕ src/cache_node.cpp (+1354 -0)
➕ src/cache_node.h (+330 -0)
📝 src/fdcache_entity.cpp (+9 -0)
📝 src/fdcache_entity.h (+2 -0)
📝 src/metaheader.cpp (+76 -24)
📝 src/metaheader.h (+4 -1)
📝 src/s3fs.cpp (+781 -572)
📝 src/s3fs_help.cpp (+2 -4)
📝 src/s3fs_threadreqs.cpp (+44 -42)
📝 src/s3fs_threadreqs.h (+4 -1)
📝 src/s3objlist.cpp (+52 -28)
📝 src/s3objlist.h (+9 -4)
📝 src/types.h (+103 -0)

📄 Description

Relevant Issue (if applicable)

#2674 #2675 #2676

Advance information

This PR is Phase 3 of fixes to refactor and improve StatCache.
(Draft until #2676 is merged. Subsequent PRs will follow based on this PR codes.)

This is the main purpose of refactorings for StatCache.
Reviewing this refactoring may take some time, so thank you in advance for your help.
(The basic idea is that the cache map in StatCache has been changed to a dedicated tree class(StatCacheNode).)

Background

Stat information/Meta headers are managed by the StatCache class, but it has become complicated due to past modifications.
Since it is difficult to implement further modifications in the current StatCache class, I decided to review the StatCache class.

Summary

Currently, StatCache caches and manages stat information and meta headers.
The cache is complicated, with regular file and directory caches, symbolic link caches, negative caches, and maps to prevent cache deletion.

So, these are implemented by new StaCacheNode base class and its derived classes, and the current StatCache has been changed to manage the StaCacheNode map.

There are no changes to the specifications in terms of functionality (support for files, directories, and symbolic links, expiration date, cache out block, negative cache, etc.).

Major changes

StatCache class

As before, this is a singleton cache management object and serves as an accessor to cached data.
Previous cache types(file, directory, symbolic link, negative cache) are managed and implemented by new StatCacheNode classes.

StatCacheNode base class and derived classes

The StatCacheNode class is the base class for each cache type.
The derived classes are FileStatCache, DirStatCache, SymlinkStatCache, and NegativeStatCache.
The Truncate(cache out) and expiration Handling for each cache are also implemented in these classes.
The blocking cache out function is managed by a flag in thess classes.
As described above, the cache data operations that were previously implemented in the StatCache class are now implemented in these classes.

Structure of Cache data

Currently, the structure of cache data was path <-> cache data as a simple mapping.
(These maps are stat_cache_t, symlink_cache_t, notruncate_dir_map_t, etc.)
In the new structure, the cache data is a directory tree structure with the mount point at the top.

Cache type definition

Currently, the cache type was defined in a complicated way using dirtype(enum) for mainly type, and flags using the type of directory.
(This dues to historical reasons due to the modification.)
The new type can now be expressed only with objtype_t(enum).
This objtype_t corresponds to the each StatCacheNode derived class.
In addition, directory types (dir/, dir, dir_$folder$, <no directory entity>) are all expressed with this objtype_t.
Utility functions such as determining objtype_t variables have also been unified.

Others

The above changes to the StatCache class and the addition of the StatCacheNode class have resulted in changes to the implementation of each file. (This was mainly done in the s3fs.cpp file.)

Notes

The data managed by the StatCacheNode class (including derived classes) includes stat and meta headers.
Previously StatCache class, this data was registered in the meta header, and stat data was created and registered through internal processing.
In the new code, it can register with stat and meta, or register stat only. (meta only registration is not possible.)
Registering stat only will be used when implementing future performance tuning.(Currently, we are not using it yet)

About s3proxy

@gaul
Due to the s3proxy specifications, some code changes were required.
As commented in the code, s3proxy responds as success to Head requests by dir path when the only dir/ object exists.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/2681 **Author:** [@ggtakec](https://github.com/ggtakec) **Created:** 6/6/2025 **Status:** ✅ Merged **Merged:** 7/24/2025 **Merged by:** [@gaul](https://github.com/gaul) **Base:** `master` ← **Head:** `stat/04_statnode` --- ### 📝 Commits (1) - [`3383fd7`](https://github.com/s3fs-fuse/s3fs-fuse/commit/3383fd73f758260688c50680784ae9f5f146edfd) Refactor to change StatCache class and add StatCacheNode classes ### 📊 Changes **16 files changed** (+3012 additions, -1394 deletions) <details> <summary>View changed files</summary> 📝 `src/Makefile.am` (+1 -0) 📝 `src/cache.cpp` (+210 -615) 📝 `src/cache.h` (+31 -103) ➕ `src/cache_node.cpp` (+1354 -0) ➕ `src/cache_node.h` (+330 -0) 📝 `src/fdcache_entity.cpp` (+9 -0) 📝 `src/fdcache_entity.h` (+2 -0) 📝 `src/metaheader.cpp` (+76 -24) 📝 `src/metaheader.h` (+4 -1) 📝 `src/s3fs.cpp` (+781 -572) 📝 `src/s3fs_help.cpp` (+2 -4) 📝 `src/s3fs_threadreqs.cpp` (+44 -42) 📝 `src/s3fs_threadreqs.h` (+4 -1) 📝 `src/s3objlist.cpp` (+52 -28) 📝 `src/s3objlist.h` (+9 -4) 📝 `src/types.h` (+103 -0) </details> ### 📄 Description ### Relevant Issue (if applicable) #2674 #2675 #2676 ### Advance information This PR is Phase 3 of fixes to refactor and improve StatCache. (Draft until #2676 is merged. Subsequent PRs will follow based on this PR codes.) This is the main purpose of refactorings for StatCache. Reviewing this refactoring may take some time, so thank you in advance for your help. (The basic idea is that the cache map in `StatCache` has been changed to a dedicated tree class(`StatCacheNode`).) ### Background Stat information/Meta headers are managed by the `StatCache` class, but it has become complicated due to past modifications. Since it is difficult to implement further modifications in the current `StatCache` class, I decided to review the `StatCache` class. ### Summary Currently, `StatCache` caches and manages stat information and meta headers. The cache is complicated, with regular file and directory caches, symbolic link caches, negative caches, and maps to prevent cache deletion. So, these are implemented by new `StaCacheNode` base class and its derived classes, and the current `StatCache` has been changed to manage the `StaCacheNode` map. There are no changes to the specifications in terms of functionality (support for files, directories, and symbolic links, expiration date, cache out block, negative cache, etc.). ### Major changes #### StatCache class As before, this is a singleton cache management object and serves as an accessor to cached data. Previous cache types(file, directory, symbolic link, negative cache) are managed and implemented by new `StatCacheNode` classes. #### StatCacheNode base class and derived classes The `StatCacheNode` class is the base class for each cache type. The derived classes are `FileStatCache`, `DirStatCache`, `SymlinkStatCache`, and `NegativeStatCache`. The Truncate(cache out) and expiration Handling for each cache are also implemented in these classes. The blocking cache out function is managed by a flag in thess classes. As described above, the cache data operations that were previously implemented in the `StatCache` class are now implemented in these classes. #### Structure of Cache data Currently, the structure of cache data was `path` <-> `cache data` as a simple mapping. (These maps are `stat_cache_t`, `symlink_cache_t`, `notruncate_dir_map_t`, etc.) In the new structure, the cache data is a directory tree structure with the mount point at the top. #### Cache type definition Currently, the cache type was defined in a complicated way using `dirtype`(enum) for mainly type, and flags using the type of directory. (This dues to historical reasons due to the modification.) The new type can now be expressed only with `objtype_t`(enum). This `objtype_t` corresponds to the each `StatCacheNode` derived class. In addition, directory types (`dir/`, `dir`, `dir_$folder$`, `<no directory entity>`) are all expressed with this `objtype_t`. Utility functions such as determining `objtype_t` variables have also been unified. #### Others The above changes to the `StatCache` class and the addition of the `StatCacheNode` class have resulted in changes to the implementation of each file. (This was mainly done in the s3fs.cpp file.) #### Notes The data managed by the `StatCacheNode` class (including derived classes) includes `stat` and `meta` headers. Previously `StatCache` class, this data was registered in the `meta` header, and `stat` data was created and registered through internal processing. In the new code, it can register with `stat` and `meta`, or register `stat` only. (`meta` only registration is not possible.) Registering `stat` only will be used when implementing future performance tuning.(Currently, we are not using it yet) #### About s3proxy @gaul Due to the s3proxy specifications, some code changes were required. As commented in the code, s3proxy responds as success to Head requests by `dir` path when the only `dir/` object exists. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>