[GH-ISSUE #2507] If there are two S3 clients deployed on different machines, and client 1 writes a file, how can client 2 read the file written by client 1 in a timely manner？ #1222

New issue

Open

opened 2026-03-04 01:52:22 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:52:22 +03:00

Owner

Originally created by @loongyiyao on GitHub (Aug 1, 2024).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2507

Additional Information
Version of s3fs being used (s3fs --version)
Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)
Kernel information (uname -r)
GNU/Linux Distribution, if applicable (cat /etc/os-release)
How to run s3fs, if applicable
/usr/bin/s3fs fangzhenyun /dev/mount1 -o url=https://video.ge.com:9000/ -o endpoint=cn-east-1 -o sigv2 -o passwd_file=/home/zxsrtn/.passwd-s3fs -o use_path_request_style -o allow_other -o umask=0 -o use_cache=/dev/cache1 -o del_cache -o ensure_diskfree=12288 -o enable_noobj_cache -o parallel_count=20 -o multipart_size=52 -o dbglevel=warn -o logfile=/home/output.log

Originally created by @loongyiyao on GitHub (Aug 1, 2024). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2507 Additional Information Version of s3fs being used (s3fs --version) Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse) Kernel information (uname -r) GNU/Linux Distribution, if applicable (cat /etc/os-release) How to run s3fs, if applicable /usr/bin/s3fs fangzhenyun /dev/mount1 -o url=https://video.ge.com:9000/ -o endpoint=cn-east-1 -o sigv2 -o passwd_file=/home/zxsrtn/.passwd-s3fs -o use_path_request_style -o allow_other -o umask=0 -o use_cache=/dev/cache1 -o del_cache -o ensure_diskfree=12288 -o enable_noobj_cache -o parallel_count=20 -o multipart_size=52 -o dbglevel=warn -o logfile=/home/output.log

kerem commented

2026-03-04 01:52:22 +03:00

Author

Owner

@gaul commented on GitHub (Aug 6, 2024):

-o stat_cache_expire (default is 900) controls the metadata cache. @ggtakec does the data cache have any kind of staleness check?

@gaul commented on GitHub (Aug 6, 2024): -o stat_cache_expire (default is 900) controls the metadata cache. @ggtakec does the data cache have any kind of staleness check?

kerem commented

2026-03-04 01:52:22 +03:00

Author

Owner

@ggtakec commented on GitHub (Aug 11, 2024):

@loongyiyao I'm sorry for my late reply.

s3fs's stat cache caches meta information for files and directories(objects).
In other words, it caches the stat information for the files themselves.
Separately from that, the list of files and subdirectories in the directory is constantly checked by s3fs.

So, if you create a new file, the file list in the directory will be updated, so if you run a command that scans the directory(ex. ls

), the new file should be found by s3fs.
If you change(update) an existing file rather than creating a new file, it will be affected by the stat cache, so s3fs can not normally notice until the cache times out.
(But if you directly operate the updated file, such as opening it, the stat information will be reread, so s3fs will be able to notice that it has been updated.)

Note that this behavior applies to files created with the s3fs, and the behavior is slightly different for files(objects) created with other clients such as the AWS CLI.

@ggtakec commented on GitHub (Aug 11, 2024): @loongyiyao I'm sorry for my late reply. s3fs's `stat cache` caches meta information for files and directories(objects). In other words, it caches the stat information for the files themselves. Separately from that, the list of files and subdirectories in the directory is constantly checked by s3fs. So, if you create a new file, the file list in the directory will be updated, so if you run a command that scans the directory(ex. ls <dir path>), the new file should be found by s3fs. If you change(update) an existing file rather than creating a new file, it will be affected by the `stat cache`, so s3fs can not normally notice until the cache times out. (But if you directly operate the updated file, such as opening it, the stat information will be reread, so s3fs will be able to notice that it has been updated.) _Note that this behavior applies to files created with the s3fs, and the behavior is slightly different for files(objects) created with other clients such as the AWS CLI._

kerem commented

2026-03-04 01:52:22 +03:00

Author

Owner

@impowerdevs commented on GitHub (Jun 11, 2025):

Hello @ggtakec ,

We have many files grouped in folders stored in directories in S3 - 200k in one directory.
This means that an ls operation could be costly.
we have some clients writing directly to S3

Issue
AFTER successful PUT to aws/S3
We have to wait 20s-120s until the file is available via s3fs using direct access (~fs read (absolute/path))

Question
Is there a way to force s3fs the access to one file ?
OR does the absolute file path access on OS level imply scanning through the inode catalogues of all directories ?

@impowerdevs commented on GitHub (Jun 11, 2025): Hello @ggtakec , 1. We have many files grouped in folders stored in directories in S3 - 200k in one directory. 1. This means that an `ls` operation could be costly. 1. we have some clients writing directly to S3 Issue AFTER successful PUT to aws/S3 We have to wait 20s-120s until the file is available via s3fs using direct access (~`fs read (absolute/path)`) Question Is there a way to force s3fs the access to one file ? OR does the absolute file path access on OS level imply scanning through the inode catalogues of all directories ?