starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 05:16:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #92] Performance issues: s3fs_create -> 19 HTTP requests! #57

New issue
Closed
opened 2026-03-04 01:41:36 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:41:36 +03:00
Owner
Copy link

Originally created by @bornmw on GitHub (Dec 3, 2014).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/92

To create a single file s3fs makes 19 HTTP calls.
That's insane.
s3fs_create makes two different calls to check_parent_object_access, each making 4 HTTP requests, that should be optimized...

Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:23 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:23 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: create zero byte file object.
Dec 3 20:33:24 ip-10-235-11-134 s3fs: uploading... [path=/100000.0.out__][fd=-1][size=0]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:24 ip-10-235-11-134 s3fs: delete stat cache entry[path=/100000.0.out__]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:24 ip-10-235-11-134 s3fs: add stat cache entry[path=/100000.0.out__]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: stat cache hit [path=/100000.0.out__][time=1417638804][hit count=0]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: stat cache hit [path=/100000.0.out__][time=1417638804][hit count=1]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: stat cache hit [path=/100000.0.out__][time=1417638804][hit count=2]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: uploading... [path=/100000.0.out__][fd=5][size=100000]
Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__
Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200
Dec 3 20:33:24 ip-10-235-11-134 s3fs: delete stat cache entry[path=/100000.0.out__]

Originally created by @bornmw on GitHub (Dec 3, 2014). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/92 To create a single file s3fs makes 19 HTTP calls. That's insane. s3fs_create makes two different calls to check_parent_object_access, each making 4 HTTP requests, that should be optimized... Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/ Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24 Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000 Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:23 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:23 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:23 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:23 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__/ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out___%24folder%24 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 404 was returned, returning ENOENT Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com?delimiter=/&prefix=100000.0.out__/&max-keys=1000 Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:24 ip-10-235-11-134 s3fs: contents_xp->nodesetval is empty. Dec 3 20:33:24 ip-10-235-11-134 s3fs: create zero byte file object. Dec 3 20:33:24 ip-10-235-11-134 s3fs: uploading... [path=/100000.0.out__][fd=-1][size=0] Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:24 ip-10-235-11-134 s3fs: delete stat cache entry[path=/100000.0.out__] Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:24 ip-10-235-11-134 s3fs: add stat cache entry[path=/100000.0.out__] Dec 3 20:33:24 ip-10-235-11-134 s3fs: stat cache hit [path=/100000.0.out__][time=1417638804][hit count=0] Dec 3 20:33:24 ip-10-235-11-134 s3fs: stat cache hit [path=/100000.0.out__][time=1417638804][hit count=1] Dec 3 20:33:24 ip-10-235-11-134 s3fs: stat cache hit [path=/100000.0.out__][time=1417638804][hit count=2] Dec 3 20:33:24 ip-10-235-11-134 s3fs: uploading... [path=/100000.0.out__][fd=5][size=100000] Dec 3 20:33:24 ip-10-235-11-134 s3fs: connecting to URL http://webdav-test.s3.amazonaws.com/100000.0.out__ Dec 3 20:33:24 ip-10-235-11-134 s3fs: HTTP response code 200 Dec 3 20:33:24 ip-10-235-11-134 s3fs: delete stat cache entry[path=/100000.0.out__]
kerem closed this issue 2026-03-04 01:41:36 +03:00
kerem commented 2026-03-04 01:41:37 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 6, 2015):

I know...
s3fs makes many http request(list bucket) because s3fs should check same name object and directory when creating new file.
For supporting directory object which is made by s3fs and old s3fs, was console, etc(folder suffix).

The listbucket request takes costs and to be lower performance, so s3fs has the cache for objects stats.

If you can, please use and try to set value for "max_stat_cache_size" and "stat_cache_expire", "enable_noobj_cache" options.
I think enable_noobj_cache option makes good performance.

Thanks in advance for your assistance.

<!-- gh-comment-id:68881139 --> @ggtakec commented on GitHub (Jan 6, 2015): I know... s3fs makes many http request(list bucket) because s3fs should check same name object and directory when creating new file. For supporting directory object which is made by s3fs and old s3fs, was console, etc(_folder_ suffix). The listbucket request takes costs and to be lower performance, so s3fs has the cache for objects stats. If you can, please use and try to set value for "max_stat_cache_size" and "stat_cache_expire", "enable_noobj_cache" options. I think enable_noobj_cache option makes good performance. Thanks in advance for your assistance.
kerem commented 2026-03-04 01:41:37 +03:00
Author
Owner
Copy link

@bornmw commented on GitHub (Jan 6, 2015):

Takeshi,

Instead of making lots of requests to pre-check for all that before creating a file, wouldn't it make sense to send just one request to try creating the file, and if it doesn't fail assume that everything is fine?

Thanks,
Oleg

On January 6, 2015 7:38:51 AM PST, Takeshi Nakatani notifications@github.com wrote:

I know...
s3fs makes many http request(list bucket) because s3fs should check
same name object and directory when creating new file.
For supporting directory object which is made by s3fs and old s3fs, was
console, etc(folder suffix).

The listbucket request takes costs and to be lower performance, so s3fs
has the cache for objects stats.

If you can, please use and try to set value for "max_stat_cache_size"
and "stat_cache_expire", "enable_noobj_cache" options.
I think enable_noobj_cache option makes good performance.

Thanks in advance for your assistance.

Reply to this email directly or view it on GitHub:
https://github.com/s3fs-fuse/s3fs-fuse/issues/92#issuecomment-68881139

Sent from my Android phone with K-9 Mail. Please excuse my brevity.

<!-- gh-comment-id:68889848 --> @bornmw commented on GitHub (Jan 6, 2015): Takeshi, Instead of making lots of requests to pre-check for all that before creating a file, wouldn't it make sense to send just one request to try creating the file, and if it doesn't fail assume that everything is fine? Thanks, Oleg On January 6, 2015 7:38:51 AM PST, Takeshi Nakatani notifications@github.com wrote: > I know... > s3fs makes many http request(list bucket) because s3fs should check > same name object and directory when creating new file. > For supporting directory object which is made by s3fs and old s3fs, was > console, etc(_folder_ suffix). > > The listbucket request takes costs and to be lower performance, so s3fs > has the cache for objects stats. > > If you can, please use and try to set value for "max_stat_cache_size" > and "stat_cache_expire", "enable_noobj_cache" options. > I think enable_noobj_cache option makes good performance. > > Thanks in advance for your assistance. > > --- > > Reply to this email directly or view it on GitHub: > https://github.com/s3fs-fuse/s3fs-fuse/issues/92#issuecomment-68881139 ## Sent from my Android phone with K-9 Mail. Please excuse my brevity.
kerem commented 2026-03-04 01:41:37 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 6, 2015):

It is the inadequate and is only determined by the successes and failures of file generation.
If there is an object of the files or directories with the same name, and in a case where permission is not allowed for that object must be checked before writing.
Check permission is a process that s3fs is doing its own, because it is not a function of S3, the process is not able to be omitted.

As a low performance of measures by a large amount of listbucket request, s3fs will have a cache of the above stats.
In addition, if you are in parallel processing listbucket request, the number can be controlled by multireq_max.

Regards,

<!-- gh-comment-id:68893589 --> @ggtakec commented on GitHub (Jan 6, 2015): It is the inadequate and is only determined by the successes and failures of file generation. If there is an object of the files or directories with the same name, and in a case where permission is not allowed for that object must be checked before writing. Check permission is a process that s3fs is doing its own, because it is not a function of S3, the process is not able to be omitted. As a low performance of measures by a large amount of listbucket request, s3fs will have a cache of the above stats. In addition, if you are in parallel processing listbucket request, the number can be controlled by multireq_max. Regards,
kerem commented 2026-03-04 01:41:37 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 17, 2016):

I'm sorry for that this issue had been left at a long period of time.

I'm closing this issue, if you have a problem yet, please post new issue or reopen this issue.

Thanks in advance for your help.

<!-- gh-comment-id:172299471 --> @ggtakec commented on GitHub (Jan 17, 2016): I'm sorry for that this issue had been left at a long period of time. I'm closing this issue, if you have a problem yet, please post new issue or reopen this issue. Thanks in advance for your help.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#57
No description provided.