[GH-ISSUE #2233] [Question] Determining if s3fs has completed syncing files to S3 #1134

New issue

Closed

opened 2026-03-04 01:51:38 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:51:38 +03:00

Owner

Originally created by @ju2wheels on GitHub (Jul 26, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2233

Hi, im trying to find more technical information on how to figure out if s3fs is done syncing any file changes to S3.

For my usage, im mounting a bucket to a folder, making some changes to the contents of that folder and then i want to unmount the s3fs mount and shutdown my instance once im sure s3fs is done syncing.

The only two methods i can think of coming up with for doing this is a combination of the following:

Check the cache directory (which I have configured to be /tmp/s3fs) for what appear to be temporary files for tracking the sync (they may be actual duplicates or sharded chunks of the files being uploaded but i am not sure) and assume its safe to unmount if none exist. I saw files of the format /tmp/s3fs/.<bucket-name>.mirror/<random file name>.tmp .
Attempt fusermount -u in a loop until it succeeds after doing the above check and waiting for all temp files to be gone (since this should fail to unmount with device/resource busy if theres any open file handles left).

This seems to be the best way i could come up with given that i cany find documentation on how to check the sync status unless i missed it somewhere, but i still feel like this potentially a sync/unmount race condition waiting to happen even if i combine the two approaches above.

Are there any other options here?

Originally created by @ju2wheels on GitHub (Jul 26, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2233 Hi, im trying to find more technical information on how to figure out if s3fs is done syncing any file changes to S3. For my usage, im mounting a bucket to a folder, making some changes to the contents of that folder and then i want to unmount the s3fs mount and shutdown my instance once im sure s3fs is done syncing. The only two methods i can think of coming up with for doing this is a combination of the following: 1. Check the cache directory (which I have configured to be /tmp/s3fs) for what appear to be temporary files for tracking the sync (they may be actual duplicates or sharded chunks of the files being uploaded but i am not sure) and assume its safe to unmount if none exist. I saw files of the format `/tmp/s3fs/.<bucket-name>.mirror/<random file name>.tmp` . 2. Attempt fusermount -u in a loop until it succeeds after doing the above check and waiting for all temp files to be gone (since this should fail to unmount with device/resource busy if theres any open file handles left). This seems to be the best way i could come up with given that i cany find documentation on how to check the sync status unless i missed it somewhere, but i still feel like this potentially a sync/unmount race condition waiting to happen even if i combine the two approaches above. Are there any other options here?

kerem closed this issue

2026-03-04 01:51:38 +03:00

kerem commented

2026-03-04 01:51:39 +03:00

Author

Owner

@gaul commented on GitHub (Jul 28, 2023):

s3fs does not asynchronously sync files. During a call to fsync or close, the object is synchronously uploaded to S3. Thus if the calls succeed then the file exists in S3.

@gaul commented on GitHub (Jul 28, 2023): s3fs does not asynchronously sync files. During a call to `fsync` or `close`, the object is synchronously uploaded to S3. Thus if the calls succeed then the file exists in S3.

kerem referenced this issue

2026-03-04 02:02:25 +03:00

[PR #1134] [MERGED] Verify file size via metadata as well as data #1842