[PR #965] [MERGED] Improvement of curl session pool for multipart #1732

New issue

Closed

opened 2026-03-04 02:01:54 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:01:54 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/965
Author: @ggtakec
Created: 2/25/2019
Status: ✅ Merged
Merged: 2/27/2019
Merged by: @ggtakec

Base: master ← Head: fix_multi_x

📝 Commits (1)

10d9f75 Improvement of curl session pool for multipart

📊 Changes

2 files changed (+267 additions, -182 deletions)

View changed files

📝 src/curl.cpp (+243 -166)
📝 src/curl.h (+24 -16)

📄 Description

Relevant Issue (if applicable)

#935

Details

Improvement of renegotiation of SSL sessions in s3fs multipart request.

Current

s3fs is pooling curl handles inside it.
This pool behaves like a LIFO.
In the multipart request of HEAD, PUT, POST, and GET, SSL re-negotiation has occurred even though s3fs uses the curl handle of pool.
The session has been disconnected from the S3 server in about 5-10 seconds.

About HEAD request

In the case of a HEAD request, s3fs may send a request with a 400/404 error in order to confirm the existence of the file.
This normally CURL closes the session.
The solution is to set the CURLOPT_KEEP_SENDING_ON_ERROR option to the CURL handle.
Also, since the S3 server does not use ALPN, you should also use the CURLOPT_SSL_ENABLE_ALPN option.

However, the curl version of OS on TravisCI used by s3fs is low, and these options are not available.
In this PR, these options were commented out and could not be used.

Thus, if s3fs receives a 400/404 error after a HEAD request, renegotiation will occur.
(If it is a directory or file created in s3fs, 400/404 does not occur, so we left it as it is.)

About Other multipart request

Currently, in the flow of s3fs multipart request (POST etc.), first create a CURL wrapped object to be executed in parallel and then use it to communicate.
s3fs allocates a CURL handle when creating these objects.
For this reason, when actually communicating (getting a semaphore), the session has already been disconnected from the S3 server.
Therefore, renegotiation occurred.

With this PR, CURL handles are not allocated when creating CURL wrapped objects. It allocates it just before communication.
CURL handles are pooled inside s3fs and they can be retrieved as LIFOs.
Since it is a LIFO, sessions will never be lost if you use the handle that you used before.

This time, I also changed the pool of CURL handles.
When the pool becomes full, s3fs discards the oldest handle.

Result

SSL renegotiation does not occur with uploading muiltiart.
(Only the first time occurred)

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/965 **Author:** [@ggtakec](https://github.com/ggtakec) **Created:** 2/25/2019 **Status:** ✅ Merged **Merged:** 2/27/2019 **Merged by:** [@ggtakec](https://github.com/ggtakec) **Base:** `master` ← **Head:** `fix_multi_x` --- ### 📝 Commits (1) - [`10d9f75`](https://github.com/s3fs-fuse/s3fs-fuse/commit/10d9f75366558e110242667d84b52382f79ab632) Improvement of curl session pool for multipart ### 📊 Changes **2 files changed** (+267 additions, -182 deletions) <details> <summary>View changed files</summary> 📝 `src/curl.cpp` (+243 -166) 📝 `src/curl.h` (+24 -16) </details> ### 📄 Description ### Relevant Issue (if applicable) #935 ### Details Improvement of renegotiation of SSL sessions in s3fs multipart request. #### Current s3fs is pooling curl handles inside it. This pool behaves like a LIFO. In the multipart request of HEAD, PUT, POST, and GET, SSL re-negotiation has occurred even though s3fs uses the curl handle of pool. The session has been disconnected from the S3 server in about 5-10 seconds. #### About HEAD request In the case of a HEAD request, s3fs may send a request with a 400/404 error in order to confirm the existence of the file. This normally CURL closes the session. The solution is to set the CURLOPT_KEEP_SENDING_ON_ERROR option to the CURL handle. Also, since the S3 server does not use ALPN, you should also use the CURLOPT_SSL_ENABLE_ALPN option. However, the curl version of OS on TravisCI used by s3fs is low, and these options are not available. In this PR, these options were commented out and could not be used. Thus, if s3fs receives a 400/404 error after a HEAD request, renegotiation will occur. (If it is a directory or file created in s3fs, 400/404 does not occur, so we left it as it is.) #### About Other multipart request Currently, in the flow of s3fs multipart request (POST etc.), first create a CURL wrapped object to be executed in parallel and then use it to communicate. s3fs allocates a CURL handle when creating these objects. For this reason, when actually communicating (getting a semaphore), the session has already been disconnected from the S3 server. Therefore, renegotiation occurred. With this PR, CURL handles are not allocated when creating CURL wrapped objects. It allocates it just before communication. CURL handles are pooled inside s3fs and they can be retrieved as LIFOs. Since it is a LIFO, sessions will never be lost if you use the handle that you used before. This time, I also changed the pool of CURL handles. When the pool becomes full, s3fs discards the oldest handle. #### Result SSL renegotiation does not occur with uploading muiltiart. (Only the first time occurred) --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>