starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #935] readdir performance regression since 1.84 #530

New issue

Closed

opened 2026-03-04 01:46:25 +03:00 by kerem · 10 comments

kerem commented

2026-03-04 01:46:25 +03:00

Owner

Originally created by @gaul on GitHub (Jan 28, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/935

Using the goofys benchmark script, I observe much slower ls_files performance in master than 1.84 (results are time, lower is better):

1.84:

$ ~/work/go-workspace/src/github.com/kahing/goofys/bench/bench.sh '/home/gaul/work/s3fs-fuse/src/s3fs gaultest-us-west-1 /home/gaul/mnt /home/gaul/mnt ls
ls_files 2.913
ls_files 0.985
ls_files 0.829
ls_files 0.852
ls_files 0.808
ls_files 0.811
ls_files 1.093
ls_files 0.959
ls_files 0.933
ls_files 0.840

master:

$ ~/work/go-workspace/src/github.com/kahing/goofys/bench/bench.sh '/home/gaul/work/s3fs-fuse/src/s3fs gaultest-us-west-1 /home/gaul/mnt' /home/gaul/mnt ls
ls_files 16.186
ls_files 9.720
ls_files 11.963
ls_files 12.418
ls_files 10.156
ls_files 10.072
ls_files 9.872
ls_files 9.751
ls_files 9.615
ls_files 10.418

We should analyze this before releasing 1.85. The script reveals other interesting results including improvements to bulk read and write performance.

Originally created by @gaul on GitHub (Jan 28, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/935 Using the goofys benchmark script, I observe much slower `ls_files` performance in master than 1.84 (results are time, lower is better): 1.84: ``` $ ~/work/go-workspace/src/github.com/kahing/goofys/bench/bench.sh '/home/gaul/work/s3fs-fuse/src/s3fs gaultest-us-west-1 /home/gaul/mnt /home/gaul/mnt ls ls_files 2.913 ls_files 0.985 ls_files 0.829 ls_files 0.852 ls_files 0.808 ls_files 0.811 ls_files 1.093 ls_files 0.959 ls_files 0.933 ls_files 0.840 ``` master: ``` $ ~/work/go-workspace/src/github.com/kahing/goofys/bench/bench.sh '/home/gaul/work/s3fs-fuse/src/s3fs gaultest-us-west-1 /home/gaul/mnt' /home/gaul/mnt ls ls_files 16.186 ls_files 9.720 ls_files 11.963 ls_files 12.418 ls_files 10.156 ls_files 10.072 ls_files 9.872 ls_files 9.751 ls_files 9.615 ls_files 10.418 ``` We should analyze this before releasing 1.85. The script reveals other interesting results including improvements to bulk read and write performance.

kerem closed this issue

2026-03-04 01:46:25 +03:00

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@gaul commented on GitHub (Jan 28, 2019):

46d79c5bc2 introduced the regression, which improved readdir times in my tests.

@gaul commented on GitHub (Jan 28, 2019): 46d79c5bc275c1df8cdd4e2ca0884efc68247e77 introduced the regression, which improved `readdir` times in my tests.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 28, 2019):

@gaul Thanks for the report.
I will concentrate and check it this week.

@ggtakec commented on GitHub (Jan 28, 2019): @gaul Thanks for the report. I will concentrate and check it this week.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@gaul commented on GitHub (Jan 28, 2019):

The benchmark script may have an issue. I observe a 25% improvement when running the same test manually:

without 46d79c5bc2:

$ time ls mnt/dir/ | wc -l
1000

real    0m8.109s
user    0m0.001s
sys     0m0.003s

with 46d79c5bc2:

$ time ls mnt/dir/ | wc -l
1000

real    0m5.715s
user    0m0.007s
sys     0m0.005s

@gaul commented on GitHub (Jan 28, 2019): The benchmark script may have an issue. I observe a 25% improvement when running the same test manually: without 46d79c5bc275c1df8cdd4e2ca0884efc68247e77: ``` $ time ls mnt/dir/ | wc -l 1000 real 0m8.109s user 0m0.001s sys 0m0.003s ``` with 46d79c5bc275c1df8cdd4e2ca0884efc68247e77: ``` $ time ls mnt/dir/ | wc -l 1000 real 0m5.715s user 0m0.007s sys 0m0.005s ```

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@gaul commented on GitHub (Jan 29, 2019):

It appears 46d79c5bc2 increases CPU usage partially from malloc_trim. Still investigating...

@gaul commented on GitHub (Jan 29, 2019): It appears 46d79c5bc275c1df8cdd4e2ca0884efc68247e77 increases CPU usage partially from `malloc_trim`. Still investigating...

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@gaul commented on GitHub (Feb 6, 2019):

I suspect that s3fs is not reusing HTTPS connections and the CPU overhead comes from the initial handshake. Using HTTP does not show this regression.

@gaul commented on GitHub (Feb 6, 2019): I suspect that s3fs is not reusing HTTPS connections and the CPU overhead comes from the initial handshake. Using HTTP does not show this regression.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 11, 2019):

@gaul I confirmed that the session was not reused and HTTPS(TLS) handshake was running.
I will immediately check the contents of the code.

@ggtakec commented on GitHub (Feb 11, 2019): @gaul I confirmed that the session was not reused and HTTPS(TLS) handshake was running. I will immediately check the contents of the code.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 17, 2019):

The session with S3 has been disconnected from the server in around 5 seconds and it is CLOSE_WAIT.
Now I can not control this with the CURL option.
(I tried the CURLOPT_TCP_KEEPALIVE option etc.)
I will examine it in succession.

@ggtakec commented on GitHub (Feb 17, 2019): The session with S3 has been disconnected from the server in around 5 seconds and it is CLOSE_WAIT. Now I can not control this with the CURL option. (I tried the CURLOPT_TCP_KEEPALIVE option etc.) I will examine it in succession.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 17, 2019):

One reason why the socket is closed was that we received 400 errors in the HEAD request(ex. ls < dir > ).
In case of 400 error, "Connection: close" is received and closed with libcurl.
By specifying CURLOPT_KEEP_SENDING_ON_ERROR by curl_easy_setopt, closing at HEAD request seems to be avoidable.
I'll continue to check about multipart upload.

@ggtakec commented on GitHub (Feb 17, 2019): One reason why the socket is closed was that we received 400 errors in the HEAD request(ex. ls < dir > ). In case of 400 error, "Connection: close" is received and closed with libcurl. By specifying CURLOPT_KEEP_SENDING_ON_ERROR by curl_easy_setopt, closing at HEAD request seems to be avoidable. I'll continue to check about multipart upload.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 27, 2019):

I merged #965 .

It does not set the CURLOPT_KEEP_SENDING_ON_ERROR option.
So, renegotiation of SSL may occur at HEAD request.

However, I reviewed the pool of curl handles with that PR and reviewed the processing of multipart request.
As a result, renegotiation of SSL does not occur in multipart POST/GET.

Please see #965 comments for detail.

@ggtakec commented on GitHub (Feb 27, 2019): I merged #965 . It does not set the CURLOPT_KEEP_SENDING_ON_ERROR option. So, renegotiation of SSL may occur at HEAD request. However, I reviewed the pool of curl handles with that PR and reviewed the processing of multipart request. As a result, renegotiation of SSL does not occur in multipart POST/GET. Please see #965 comments for detail.

kerem commented

2026-03-04 01:46:26 +03:00

Author

Owner

@gaul commented on GitHub (Mar 9, 2019):

Fixed by #965.

@gaul commented on GitHub (Mar 9, 2019): Fixed by #965.