[GH-ISSUE #824] Explanation of async vs sync #480

New issue

Closed

opened 2026-03-04 01:45:58 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:45:58 +03:00

Owner

Originally created by @CMCDragonkai on GitHub (Sep 20, 2018).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/824

In the man page, it says that async and sync are all supported. I was hoping to find out how does s3fs implement async writes. I have a situation where I have a process that wants to write hundreds of thousands of files that are each a couple megabytes in size.

I was worried that if sync was the default, then the writing process will be blocked by sending an HTTP PUT to S3. On the other hand, if async was switched on, I couldn't find any information on how the writeback buffer is implemented. Is it buffering into RAM? Can these writes be spilled onto swap/disk? What about backpressure, is it possible that eventually writes will block once the buffer is full?

I also found that there is a cache option and a ensure_diskfree option, which appears to allow the possibility of using a local filesystem as both the read cache and write cache. But how does this feature interact with sync vs async mounting options?

Originally created by @CMCDragonkai on GitHub (Sep 20, 2018). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/824 In the man page, it says that async and sync are all supported. I was hoping to find out how does s3fs implement async writes. I have a situation where I have a process that wants to write hundreds of thousands of files that are each a couple megabytes in size. I was worried that if sync was the default, then the writing process will be blocked by sending an HTTP PUT to S3. On the other hand, if async was switched on, I couldn't find any information on how the writeback buffer is implemented. Is it buffering into RAM? Can these writes be spilled onto swap/disk? What about backpressure, is it possible that eventually writes will block once the buffer is full? I also found that there is a cache option and a `ensure_diskfree` option, which appears to allow the possibility of using a local filesystem as both the read cache and write cache. But how does this feature interact with sync vs async mounting options?

kerem closed this issue

2026-03-04 01:45:59 +03:00

kerem commented

2026-03-04 01:45:59 +03:00

Author

Owner

@gaul commented on GitHub (Oct 10, 2020):

Asynchronous has a different meaning here, related to whether an interrupt like SIGINT will interrupt a network call like write when the server is not responding:

https://unix.stackexchange.com/questions/146620/difference-between-sync-and-async-mount-options

This dates back to NFS which would retry operations forever. s3fs actually will return EIO after a configurable number of retries.

As for the asynchronous meaning you mean, s3fs has close-to-open semantics. Internally it spools writes to a local file and does a PUT during close or fsync. This means that the write is not complete or observable to other S3 clients until these calls complete. #367 discusses a write-behind caching system although this has numerous pitfalls and may never be implemented.

@gaul commented on GitHub (Oct 10, 2020): Asynchronous has a different meaning here, related to whether an interrupt like `SIGINT` will interrupt a network call like write when the server is not responding: https://unix.stackexchange.com/questions/146620/difference-between-sync-and-async-mount-options This dates back to NFS which would retry operations forever. s3fs actually will return `EIO` after a configurable number of retries. As for the asynchronous meaning you mean, s3fs has close-to-open semantics. Internally it spools writes to a local file and does a PUT during `close` or `fsync`. This means that the write is not complete or observable to other S3 clients until these calls complete. #367 discusses a write-behind caching system although this has numerous pitfalls and may never be implemented.

kerem referenced this issue

2026-03-04 02:00:52 +03:00

[PR #488] [MERGED] Fixed searching Content-Length without case sensitive - #480 #1523