starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #399] Does file creation really fail because of eventual consistency? #211

New issue

Closed

opened 2026-03-04 01:43:17 +03:00 by kerem · 4 comments

kerem commented

2026-03-04 01:43:17 +03:00

Owner

Originally created by @plufz on GitHub (Apr 19, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/399

Hi!

I'm trying to understand this from the documentation "Due to S3's "eventual consistency" limitations file creation can and will occasionally fail.".

Is this really true? From what I understand of "eventual consistency" file writes never fails, they just take time to be available for read.

Will files I write to my s3 buckets fail or will it just take some time before I can read the files?

Originally created by @plufz on GitHub (Apr 19, 2016). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/399 Hi! I'm trying to understand this from the documentation "Due to S3's "eventual consistency" limitations file creation can and will occasionally fail.". Is this really true? From what I understand of "eventual consistency" file writes never fails, they just take time to be available for read. Will files I write to my s3 buckets fail or will it just take some time before I can read the files?

kerem

2026-03-04 01:43:17 +03:00

closed this issue
added the
question
label

kerem commented

2026-03-04 01:43:18 +03:00

Author

Owner

@ggtakec commented on GitHub (May 14, 2016):

@plufz
This is the content that describes "eventual consistency" of the S3.
You can find "Amazon S3 Data Consistency Model" in http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html
In other words, this is the sense that "eventual consistency" of s3fs is influenced by the S3.

I do not think that there is a large time lag on my experience.
I think, as long as the AWS can not explicitly a time lag, it is not know when not to try in your environment.

Regards,

@ggtakec commented on GitHub (May 14, 2016): @plufz This is the content that describes "eventual consistency" of the S3. You can find "Amazon S3 Data Consistency Model" in http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html In other words, this is the sense that "eventual consistency" of s3fs is influenced by the S3. I do not think that there is a large time lag on my experience. I think, as long as the AWS can not explicitly a time lag, it is not know when not to try in your environment. Regards,

kerem commented

2026-03-04 01:43:18 +03:00

Author

Owner

@gaul commented on GitHub (May 14, 2016):

@plufz You can test this yourself using the eventual consistency middleware of S3Proxy:

https://github.com/andrewgaul/s3proxy/wiki/Middleware---eventual-consistency

I found that creating a file can fail because s3fs_create calls create_file_object which PUTs the object then immediately calls get_object_attribute which HEADs the object. S3Proxy models the most pessimistic form of eventual consistency, read-after-create. Only Amazon us-standard region with the default endpoint exhibits these semantics; other regions, including us-standard with the Virginia endpoint, do not. Deleting, overwriting, or updating a file's metadata can exhibit eventually consistent behavior in all regions. I explored eventual consistency semantics here:

https://github.com/andrewgaul/are-we-consistent-yet

@ggtakec s3fs would benefit from removing excessive stat calls, including in the s3fs_create function mentioned above. This would improve performance and reduce the likelihood of users observing eventual consistency.

@gaul commented on GitHub (May 14, 2016): @plufz You can test this yourself using the eventual consistency middleware of S3Proxy: https://github.com/andrewgaul/s3proxy/wiki/Middleware---eventual-consistency I found that creating a file can fail because `s3fs_create` calls `create_file_object` which PUTs the object then immediately calls `get_object_attribute` which HEADs the object. S3Proxy models the most pessimistic form of eventual consistency, read-after-create. Only Amazon us-standard region with the default endpoint exhibits these semantics; other regions, including us-standard with the Virginia endpoint, do not. Deleting, overwriting, or updating a file's metadata can exhibit eventually consistent behavior in all regions. I explored eventual consistency semantics here: https://github.com/andrewgaul/are-we-consistent-yet @ggtakec s3fs would benefit from removing excessive stat calls, including in the `s3fs_create` function mentioned above. This would improve performance and reduce the likelihood of users observing eventual consistency.

kerem commented

2026-03-04 01:43:18 +03:00

Author

Owner

@ggtakec commented on GitHub (May 15, 2016):

@andrewgaul Thanks for your help.
Although it is certainly redundant, I think that stat read(HEAD request) after the file creation(PUT) is required for confirmation of file initial creation.
But I will survey that it should be deleted in the future.

@ggtakec commented on GitHub (May 15, 2016): @andrewgaul Thanks for your help. Although it is certainly redundant, I think that stat read(HEAD request) after the file creation(PUT) is required for confirmation of file initial creation. But I will survey that it should be deleted in the future.

kerem commented

2026-03-04 01:43:19 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 30, 2019):

We kept this issue open for a long time.
I will close this, but if the problem persists, please reopen or post a new issue.

@ggtakec commented on GitHub (Mar 30, 2019): We kept this issue open for a long time. I will close this, but if the problem persists, please reopen or post a new issue.

kerem referenced this issue

2026-03-04 01:53:44 +03:00

[PR #211] [MERGED] Updated ChangeLog and configure.ac for v1.79 #1396

kerem referenced this issue

2026-03-04 01:53:45 +03:00

[PR #212] [MERGED] Updates #1399