[GH-ISSUE #1808] s3fs benchmark testing tools #924

New issue

Closed

opened 2026-03-04 01:49:58 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:49:58 +03:00

Owner

Originally created by @syu-lk4b on GitHub (Nov 18, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1808

We are trying check the performance compare with other storage , do we have a recommend benchmark testing tools . I am currently using "https://github.com/axboe/fio" ,but maybe this is not a correct tools. Basically we are trying to use s3fs-fuse to mount the s3 bucket for Machine learning training. We have enabled the local cache , but after almost everything get cached, there are still some gaps ,compare with local storage, even everything is already cached in local storage

Originally created by @syu-lk4b on GitHub (Nov 18, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1808 We are trying check the performance compare with other storage , do we have a recommend benchmark testing tools . I am currently using "https://github.com/axboe/fio" ,but maybe this is not a correct tools. Basically we are trying to use s3fs-fuse to mount the s3 bucket for Machine learning training. We have enabled the local cache , but after almost everything get cached, there are still some gaps ,compare with local storage, even everything is already cached in local storage

kerem closed this issue

2026-03-04 01:49:58 +03:00

kerem commented

2026-03-04 01:49:59 +03:00

Author

Owner

@gaul commented on GitHub (Nov 20, 2021):

s3fs does not focus on performance but we look at the goofys benchmarks since this is a high-quality implementation. Note that its published benchmarks are 2 years out of date and don't reflect the current s3fs performance: kahing/goofys#647. It would be great if you could publish a blog post comparing s3fs, goofys, or other FUSE filesystems using fio, goofys benchmark, or something else!

Local storage will always be faster that s3fs even when the cache is working. The local filesystem uses the kernel cache while s3fs uses a user-mode cache so there are overheads. That said there might be inefficiencies if you can point to some specific case that caching does not work as you expect.

@gaul commented on GitHub (Nov 20, 2021): s3fs does not focus on performance but we look at the [goofys benchmarks](https://github.com/kahing/goofys) since this is a high-quality implementation. Note that its published benchmarks are 2 years out of date and don't reflect the current s3fs performance: kahing/goofys#647. It would be great if you could publish a blog post comparing s3fs, goofys, or other FUSE filesystems using fio, goofys benchmark, or something else! Local storage will always be faster that s3fs even when the cache is working. The local filesystem uses the kernel cache while s3fs uses a user-mode cache so there are overheads. That said there might be inefficiencies if you can point to some specific case that caching does not work as you expect.

kerem commented

2026-03-04 01:49:59 +03:00

Author

Owner

@syu-lk4b commented on GitHub (Nov 22, 2021):

@gaul thanks so much. I'll definitely post a blog once i finished the benchmark testing

@syu-lk4b commented on GitHub (Nov 22, 2021): @gaul thanks so much. I'll definitely post a blog once i finished the benchmark testing

kerem commented

2026-03-04 01:49:59 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 26, 2023):

Close this issue. If you still need it, please reopen it.

@ggtakec commented on GitHub (Mar 26, 2023): Close this issue. If you still need it, please reopen it.

kerem referenced this issue

2026-03-04 02:01:45 +03:00

[PR #924] [MERGED] Prefer empty over size checks #1703