[GH-ISSUE #1643] File creation issues 9 HTTP requests sequentially #859

New issue

Open

opened 2026-03-04 01:49:28 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 01:49:28 +03:00

Owner

Originally created by @gaul on GitHub (May 1, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1643

s3fs probes the for object existence sequentially which hurts performance:

HEAD /tmp/1 HTTP/1.1
HEAD /tmp/1/ HTTP/1.1
HEAD /tmp/1_%24folder%24 HTTP/1.1
GET /?delimiter=/&max-keys=2&prefix=tmp/1/ HTTP/1.1
HEAD /tmp/1 HTTP/1.1
HEAD /tmp/1/ HTTP/1.1
HEAD /tmp/1_%24folder%24 HTTP/1.1
GET /?delimiter=/&max-keys=2&prefix=tmp/1/ HTTP/1.1
PUT /tmp/1 HTTP/1.1

Setting -o notsup_compat_dir helps but s3fs can make this much faster by issuing requests in parallel. The logic in chk_dir_object_type is a bit twisty though.

Originally created by @gaul on GitHub (May 1, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1643 s3fs probes the for object existence sequentially which hurts performance: ``` HEAD /tmp/1 HTTP/1.1 HEAD /tmp/1/ HTTP/1.1 HEAD /tmp/1_%24folder%24 HTTP/1.1 GET /?delimiter=/&max-keys=2&prefix=tmp/1/ HTTP/1.1 HEAD /tmp/1 HTTP/1.1 HEAD /tmp/1/ HTTP/1.1 HEAD /tmp/1_%24folder%24 HTTP/1.1 GET /?delimiter=/&max-keys=2&prefix=tmp/1/ HTTP/1.1 PUT /tmp/1 HTTP/1.1 ``` Setting `-o notsup_compat_dir` helps but s3fs can make this much faster by issuing requests in parallel. The logic in `chk_dir_object_type` is a bit twisty though.