starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 05:16:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2354] Readdir fails for SSE-C and prefix mount #1159

New issue
Closed
opened 2026-03-04 01:51:49 +03:00 by kerem · 3 comments
kerem commented 2026-03-04 01:51:49 +03:00
Owner
Copy link

Originally created by @jstastny on GitHub (Oct 18, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2354

When using SSE-C with prefix mount on the latest master (commit 2871975), the readdir is not listing items I create inside of the mounted directory.

Repro steps:

  1. Mount like: AWSSSECKEYS=vgXEublG7JJkhfRSZT1Ze3lefY2d7j3i s3fs -f -d -o bucket=my-bucket:/honza-testing-sse,complement_stat,allow_other,uid=0,gid=100,use_cache=/tmp/s3fs-cache,del_cache,ensure_diskfree=10000,curldbg,use_sse=custom /mnt/s3
  2. Create a file inside of the mount directory: touch /mnt/s3/aaa (the object correctly gets created in S3, including SSE-C)
  3. List the contents of the mount directory: ls /mnt/s3
  4. This gives nothing

I investigated the HTTP calls and think this is caused by the incorrect path in the retry of the HEAD request (my understanding is that the retry is intentional, because the first request happens without SSE-C headers and the retry callback contains the SSE-C headers, based on this comment).

The problem is that the path gets broken in the second attempt and the prefix (/honza-testing-sse in my case) is repeated twice there. Here are the snippets from the logs (I've changed the bucket name and removed the boring parts like SSL bundle resolution):

First HEAD request -- correct path, but missing SSE-C headers

2023-10-18T14:35:41.875Z [INF]   s3fs.cpp:readdir_multi_head(3262): [path=/][list=0]
2023-10-18T14:35:41.875Z [INF]       curl.cpp:PreHeadRequest(3208): [tpath=/aaa][bpath=aaa][save=/aaa][sseckeypos=18446744073709551615]
2023-10-18T14:35:41.875Z [INF]       curl_util.cpp:prepare_url(210): URL is https://s3.amazonaws.com/my-bucket/honza-testing-sse/aaa
2023-10-18T14:35:41.875Z [INF]       curl_util.cpp:prepare_url(243): URL changed is https://my-bucket.s3.amazonaws.com/honza-testing-sse/aaa
2023-10-18T14:35:41.875Z [INF]       curl_multi.cpp:Request(324): [count=1]
2023-10-18T14:35:41.875Z [INF]       curl.cpp:insertV4Headers(2840): computing signature [HEAD] [/honza-testing-sse/aaa] [] []
2023-10-18T14:35:41.875Z [INF]       curl_util.cpp:url_to_host(265): url is https://s3.amazonaws.com
...
2023-10-18T14:35:41.875Z [CURL DBG] > HEAD /honza-testing-sse/aaa HTTP/1.1
2023-10-18T14:35:41.875Z [CURL DBG] > Host: my-bucket.s3.amazonaws.com
2023-10-18T14:35:41.875Z [CURL DBG] > User-Agent: s3fs/1.93.2 (commit hash unknown; OpenSSL)
2023-10-18T14:35:41.875Z [CURL DBG] > Accept: */*
...
2023-10-18T14:35:41.875Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-10-18T14:35:41.875Z [CURL DBG] > x-amz-date: 20231018T143541Z
2023-10-18T14:35:41.875Z [CURL DBG] >
...
2023-10-18T14:35:41.977Z [CURL DBG] < HTTP/1.1 400 Bad Request
2023-10-18T14:35:41.977Z [CURL DBG] < x-amz-request-id: 4G8GWN3G4K164XWZ
2023-10-18T14:35:41.977Z [CURL DBG] < x-amz-id-2: 7khYl3/U23hCCXP2iz+7zBlsxg7sxOJpwQYPZit0ZfoAkc3nTQt9CgFs/mfBKAckfxsIdgwVBXA=
2023-10-18T14:35:41.977Z [CURL DBG] < Content-Type: application/xml
...
2023-10-18T14:35:41.979Z [ERR] curl.cpp:RequestPerform(2526): HEAD HTTP response code 400, returning EPERM.
2023-10-18T14:35:41.979Z [WAN] curl_multi.cpp:MultiPerform(193): thread terminated with non-zero return code: -1
2023-10-18T14:35:41.979Z [WAN] curl_multi.cpp:MultiRead(228): failed a request(400: https://my-bucket.s3.amazonaws.com/honza-testing-sse/aaa)

Second HEAD request -- incorrect path (see the /honza-testing-sse/honza-testing-sse/), but SSE-C is present

2023-10-18T14:35:41.979Z [INF]       curl.cpp:PreHeadRequest(3208): [tpath=/honza-testing-sse/aaa][bpath=aaa][save=/aaa][sseckeypos=0]
2023-10-18T14:35:41.979Z [INF]       curl_util.cpp:prepare_url(210): URL is https://s3.amazonaws.com/my-bucket/honza-testing-sse/honza-testing-sse/aaa
2023-10-18T14:35:41.979Z [INF]       curl_util.cpp:prepare_url(243): URL changed is https://my-bucket.s3.amazonaws.com/honza-testing-sse/honza-testing-sse/aaa
2023-10-18T14:35:41.979Z [INF]       curl.cpp:insertV4Headers(2840): computing signature [HEAD] [/honza-testing-sse/honza-testing-sse/aaa] [] []
2023-10-18T14:35:41.979Z [INF]       curl_util.cpp:url_to_host(265): url is https://s3.amazonaws.com
2023-10-18T14:35:41.979Z [CURL DBG] * Hostname my-bucket.s3.amazonaws.com was found in DNS cache
...
2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-date: 20231018T143541Z
2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-server-side-encryption-customer-algorithm: AES256
2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-server-side-encryption-customer-key: dmdYRXVibEc3SkpraGZSU1pUMVplM2xlZlkyZDdqM2k=
2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-server-side-encryption-customer-key-md5: 5hY9TS21HM2qJz6bFpDkgg==
2023-10-18T14:35:42.363Z [CURL DBG] >
2023-10-18T14:35:42.478Z [CURL DBG] * TLSv1.2 (IN), TLS header, Supplemental data (23):
2023-10-18T14:35:42.478Z [CURL DBG] * Mark bundle as not supporting multiuse
2023-10-18T14:35:42.478Z [CURL DBG] < HTTP/1.1 404 Not Found
2023-10-18T14:35:42.478Z [CURL DBG] < x-amz-request-id: GAWNE0Z9CZHHJM7K
2023-10-18T14:35:42.478Z [CURL DBG] < x-amz-id-2: 69CRu9zdCoqb958dahFGx4pNIU7mGGPsNPEvDgV13StKV1gRjEbmuK7GzzEORNcAeKvbIlbG5Nc=
2023-10-18T14:35:42.478Z [CURL DBG] < Content-Type: application/xml
2023-10-18T14:35:42.478Z [CURL DBG] < Date: Wed, 18 Oct 2023 14:35:41 GMT
2023-10-18T14:35:42.479Z [CURL DBG] < Server: AmazonS3
2023-10-18T14:35:42.479Z [CURL DBG] <
2023-10-18T14:35:42.479Z [INF]       curl.cpp:RequestPerform(2540): HTTP response code 404 was returned, returning ENOENT

I tested the same workflow without using the mount prefix and everything worked just fine.

Additional Information

Version of s3fs being used (s3fs --version)

V1.93 2871975

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

6.9.9-5ubuntu3

Kernel information (uname -r)

7.15.0-1037-aws (running in Docker container in Ubuntu image)

GNU/Linux Distribution, if applicable (cat /etc/os-release)

How to run s3fs, if applicable

[x] command line
[] /etc/fstab

Originally created by @jstastny on GitHub (Oct 18, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2354 When using SSE-C with prefix mount on the latest `master` (commit `2871975`), the readdir is not listing items I create inside of the mounted directory. Repro steps: 1. Mount like: `AWSSSECKEYS=vgXEublG7JJkhfRSZT1Ze3lefY2d7j3i s3fs -f -d -o bucket=my-bucket:/honza-testing-sse,complement_stat,allow_other,uid=0,gid=100,use_cache=/tmp/s3fs-cache,del_cache,ensure_diskfree=10000,curldbg,use_sse=custom /mnt/s3` 2. Create a file inside of the mount directory: `touch /mnt/s3/aaa` (the object correctly gets created in S3, including SSE-C) 3. List the contents of the mount directory: `ls /mnt/s3` 4. This gives nothing I investigated the HTTP calls and think this is caused by the incorrect path in the retry of the HEAD request (my understanding is that the retry is intentional, because the first request happens without SSE-C headers and the retry callback contains the SSE-C headers, based on [this comment](https://github.com/s3fs-fuse/s3fs-fuse/blob/2871975d1eb852f3aeb3f2a88faf2be5b920bf36/src/s3fs.cpp#L3310-L3311)). The problem is that the path gets broken in the second attempt and the prefix (`/honza-testing-sse` in my case) is repeated twice there. Here are the snippets from the logs (I've changed the bucket name and removed the boring parts like SSL bundle resolution): First `HEAD` request -- correct path, but missing SSE-C headers ``` 2023-10-18T14:35:41.875Z [INF] s3fs.cpp:readdir_multi_head(3262): [path=/][list=0] 2023-10-18T14:35:41.875Z [INF] curl.cpp:PreHeadRequest(3208): [tpath=/aaa][bpath=aaa][save=/aaa][sseckeypos=18446744073709551615] 2023-10-18T14:35:41.875Z [INF] curl_util.cpp:prepare_url(210): URL is https://s3.amazonaws.com/my-bucket/honza-testing-sse/aaa 2023-10-18T14:35:41.875Z [INF] curl_util.cpp:prepare_url(243): URL changed is https://my-bucket.s3.amazonaws.com/honza-testing-sse/aaa 2023-10-18T14:35:41.875Z [INF] curl_multi.cpp:Request(324): [count=1] 2023-10-18T14:35:41.875Z [INF] curl.cpp:insertV4Headers(2840): computing signature [HEAD] [/honza-testing-sse/aaa] [] [] 2023-10-18T14:35:41.875Z [INF] curl_util.cpp:url_to_host(265): url is https://s3.amazonaws.com ... 2023-10-18T14:35:41.875Z [CURL DBG] > HEAD /honza-testing-sse/aaa HTTP/1.1 2023-10-18T14:35:41.875Z [CURL DBG] > Host: my-bucket.s3.amazonaws.com 2023-10-18T14:35:41.875Z [CURL DBG] > User-Agent: s3fs/1.93.2 (commit hash unknown; OpenSSL) 2023-10-18T14:35:41.875Z [CURL DBG] > Accept: */* ... 2023-10-18T14:35:41.875Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-10-18T14:35:41.875Z [CURL DBG] > x-amz-date: 20231018T143541Z 2023-10-18T14:35:41.875Z [CURL DBG] > ... 2023-10-18T14:35:41.977Z [CURL DBG] < HTTP/1.1 400 Bad Request 2023-10-18T14:35:41.977Z [CURL DBG] < x-amz-request-id: 4G8GWN3G4K164XWZ 2023-10-18T14:35:41.977Z [CURL DBG] < x-amz-id-2: 7khYl3/U23hCCXP2iz+7zBlsxg7sxOJpwQYPZit0ZfoAkc3nTQt9CgFs/mfBKAckfxsIdgwVBXA= 2023-10-18T14:35:41.977Z [CURL DBG] < Content-Type: application/xml ... 2023-10-18T14:35:41.979Z [ERR] curl.cpp:RequestPerform(2526): HEAD HTTP response code 400, returning EPERM. 2023-10-18T14:35:41.979Z [WAN] curl_multi.cpp:MultiPerform(193): thread terminated with non-zero return code: -1 2023-10-18T14:35:41.979Z [WAN] curl_multi.cpp:MultiRead(228): failed a request(400: https://my-bucket.s3.amazonaws.com/honza-testing-sse/aaa) ``` Second `HEAD` request -- incorrect path (see the `/honza-testing-sse/honza-testing-sse/`), but SSE-C is present ``` 2023-10-18T14:35:41.979Z [INF] curl.cpp:PreHeadRequest(3208): [tpath=/honza-testing-sse/aaa][bpath=aaa][save=/aaa][sseckeypos=0] 2023-10-18T14:35:41.979Z [INF] curl_util.cpp:prepare_url(210): URL is https://s3.amazonaws.com/my-bucket/honza-testing-sse/honza-testing-sse/aaa 2023-10-18T14:35:41.979Z [INF] curl_util.cpp:prepare_url(243): URL changed is https://my-bucket.s3.amazonaws.com/honza-testing-sse/honza-testing-sse/aaa 2023-10-18T14:35:41.979Z [INF] curl.cpp:insertV4Headers(2840): computing signature [HEAD] [/honza-testing-sse/honza-testing-sse/aaa] [] [] 2023-10-18T14:35:41.979Z [INF] curl_util.cpp:url_to_host(265): url is https://s3.amazonaws.com 2023-10-18T14:35:41.979Z [CURL DBG] * Hostname my-bucket.s3.amazonaws.com was found in DNS cache ... 2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-date: 20231018T143541Z 2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-server-side-encryption-customer-algorithm: AES256 2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-server-side-encryption-customer-key: dmdYRXVibEc3SkpraGZSU1pUMVplM2xlZlkyZDdqM2k= 2023-10-18T14:35:42.363Z [CURL DBG] > x-amz-server-side-encryption-customer-key-md5: 5hY9TS21HM2qJz6bFpDkgg== 2023-10-18T14:35:42.363Z [CURL DBG] > 2023-10-18T14:35:42.478Z [CURL DBG] * TLSv1.2 (IN), TLS header, Supplemental data (23): 2023-10-18T14:35:42.478Z [CURL DBG] * Mark bundle as not supporting multiuse 2023-10-18T14:35:42.478Z [CURL DBG] < HTTP/1.1 404 Not Found 2023-10-18T14:35:42.478Z [CURL DBG] < x-amz-request-id: GAWNE0Z9CZHHJM7K 2023-10-18T14:35:42.478Z [CURL DBG] < x-amz-id-2: 69CRu9zdCoqb958dahFGx4pNIU7mGGPsNPEvDgV13StKV1gRjEbmuK7GzzEORNcAeKvbIlbG5Nc= 2023-10-18T14:35:42.478Z [CURL DBG] < Content-Type: application/xml 2023-10-18T14:35:42.478Z [CURL DBG] < Date: Wed, 18 Oct 2023 14:35:41 GMT 2023-10-18T14:35:42.479Z [CURL DBG] < Server: AmazonS3 2023-10-18T14:35:42.479Z [CURL DBG] < 2023-10-18T14:35:42.479Z [INF] curl.cpp:RequestPerform(2540): HTTP response code 404 was returned, returning ENOENT ``` I tested the same workflow without using the mount prefix and everything worked just fine. ### Additional Information #### Version of s3fs being used (`s3fs --version`) V1.93 2871975 #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 6.9.9-5ubuntu3 #### Kernel information (`uname -r`) 7.15.0-1037-aws (running in Docker container in Ubuntu image) #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) <!-- command result --> #### How to run s3fs, if applicable <!-- Describe the s3fs "command line" or "/etc/fstab" entry used. --> [x] command line [] /etc/fstab
kerem closed this issue 2026-03-04 01:51:49 +03:00
kerem commented 2026-03-04 01:51:50 +03:00
Author
Owner
Copy link

@jstastny commented on GitHub (Oct 18, 2023):

I have prepared an extremely naive fix (which likely breaks many other things).
Having debugged this for some time, I believe that the problem is that that retry instance of S3fsCurl takes the s3fscurl->GetPath() from the previous instance, but the path in the previous instance already has the mount_prefix prepended by get_realpath().

<!-- gh-comment-id:1769262675 --> @jstastny commented on GitHub (Oct 18, 2023): I have prepared an extremely naive [fix](https://github.com/jstastny/s3fs-fuse/commit/b5ca2c76ccdefdfef1d7c324d5e0d7deed5c4cf0) (which likely breaks many other things). Having debugged this for some time, I believe that the problem is that that retry instance of `S3fsCurl` takes the `s3fscurl->GetPath()` from the previous instance, but the `path` in the previous instance already has the `mount_prefix` [prepended](https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/curl.cpp#L3219) by [`get_realpath()`](https://github.com/s3fs-fuse/s3fs-fuse/blob/master/src/s3fs_util.cpp#L53).
kerem commented 2026-03-04 01:51:50 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 5, 2024):

@jstastny
I looked into this bug you detected.
Your fix was mostly correct, but it was needed a little more modification.

So, I created a PR as #2406 to fix this.
I have manually checked the HEAD request retries and confirmed that it works, but I would appreciate it if you could confirm this if possible.

<!-- gh-comment-id:1927069310 --> @ggtakec commented on GitHub (Feb 5, 2024): @jstastny I looked into this bug you detected. Your fix was mostly correct, but it was needed a little more modification. So, I created a PR as #2406 to fix this. I have manually checked the HEAD request retries and confirmed that it works, but I would appreciate it if you could confirm this if possible.
kerem commented 2026-03-04 01:51:50 +03:00
Author
Owner
Copy link

@jstastny commented on GitHub (Feb 5, 2024):

@ggtakec -- thanks a lot for looking into this.
@chudyandrej -- could you test #2406 in Deepnote?

<!-- gh-comment-id:1927117261 --> @jstastny commented on GitHub (Feb 5, 2024): @ggtakec -- thanks a lot for looking into this. @chudyandrej -- could you test #2406 in Deepnote?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1159
No description provided.