[PR #2399] [MERGED] fix: List Bucket/IAM Edge Cases #2519

New issue

Closed

opened 2026-03-04 02:05:57 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:05:57 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/2399
Author: @jmcarpenter2
Created: 1/23/2024
Status: ✅ Merged
Merged: 2/3/2024
Merged by: @ggtakec

Base: master ← Head: jmc/fix-list-bucket

📝 Commits (1)

bf85b38 fix: ListBucket edge cases

📊 Changes

1 file changed (+2 additions, -3 deletions)

View changed files

📝 src/s3fs.cpp (+2 -3)

📄 Description

This pull request resolves an edge case when the XML fails to add objects to the ListBucket response, and correctly identifies that there are still objects under the path provided. This can be caused by slightly restricted (but still correct) IAM permissions, where you add a condition on the s3:prefix to the s3:ListBucket permissions. Specifically, this issue only occurs in special cases where there are special characters in the prefix/object path.

Relevant Issue (if applicable)

https://github.com/s3fs-fuse/s3fs-fuse/issues/2129#issuecomment-1889763779

Details

There are two changes in this pull request.

We apply a prefix to the reiter string before checking if reiter is an empty directory. This resolves an Access Denied response from S3, because for some reason the full path did not include the / separator between the base prefix and the rest of the path.
We handle the case where directory_empty function returns -ENOTEMPTY, which implies that while adding objects from the XML response of the ListBucket CURL command, it failed but there are still objects under the path provided. This failure mode appears to happen with the special IAM policy condition mentioned above in combination with special characters in the object or prefix path. By handling this special case we getStatCacheData from the updated dirpath instead and proceed with the s3 prefix appearing in the mount.

Demonstration of fix

Basic setup
Version of s3fs being used (s3fs --version)
1.93

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)
2.9.9

Kernel information (uname -r)
4.14.334-252.552.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Toy example, based on real issue

How to run s3fs, if applicable

s3fs "david-bucket:/home/David" "/home/jovyan/work/personal" -o sigv4,iam_role=auto -o endpoint=us-east-2 -o url=https://s3.us-east-2.amazonaws.com \
    -o allow_other -o uid=1000,gid=1000 -o umask="0000" \
    -o compat_dir,complement_stat,nonempty -o dbglevel=warn -f &

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

2024-01-12T18:14:02.304Z [ERR] s3fs.cpp:list_bucket(3402): ListBucketRequest returns with error.
2024-01-12T18:14:02.304Z [ERR] s3fs.cpp:directory_empty(1259): list_bucket returns error.
2024-01-12T18:14:02.304Z [WAN] s3fs.cpp:readdir_multi_head(3299): david_stuff/ object does not have any object under it(errno=-1),
2024-01-12T18:14:02.309Z [ERR] curl.cpp:RequestPerform(2566): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>XXX</RequestId><HostId>XXX</HostId></Error>

IAM Policy

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "bucket0nav",
			"Action": [
				"s3:ListBucket"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::david-bucket"
			],
			"Condition": {
				"StringEquals": {
					"s3:prefix": [
						"",
						"home",
						"home/David"
					],
					"s3:delimiter": [
						"/"
					]
				}
			}
		},
		{
			"Sid": "bucket0recurse",
			"Action": [
				"s3:ListBucket"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::david-bucket"
			],
			"Condition": {
				"StringLike": {
					"s3:prefix": [
						"home/David/*"
					]
				}
			}
		},
		{
			"Sid": "writeaccess",
			"Action": [
				"s3:AbortMultipartUpload",
				"s3:DeleteObject",
				"s3:GetObject",
				"s3:GetObjectVersion",
				"s3:ListMultipartUploadParts",
				"s3:PutObject"
			],
			"Effect": "Allow",
			"Resource": [
				"arn:aws:s3:::david-bucket/home/David/*",
			]
		}
	]
}

We are trying to list bucket which has contents like the following

home
|------David
           |------david_stuff
                      |-------Life and Health History Survey-responses.json
                      |-------file_2.txt
                      |-------etc..

Details
It appears that the spaces in the object key Life and Health History Survey-responses.json under the david_stuff prefix is causing issues somehow in relationship with the listbucket permissions on home/David/*

To demonstrate how this fixes the issue, elaborating here on the failure mode workflow, given the example setup:

Mount mountpoint david-bucket:/home/David to /home/jovyan/work/personal, with prefix under it david_stuff/
function readdir_multi_head invoked on mount prefix /home/David
function directory_empty invoked on path david_stuff
Inside of function list_bucket, append_objects_from_xml is invoked
append_objects_from_xml_xe is invoked, trying to parse the Contents/Key or the CommonPrefixes/Prefix
- here we get contents_xp ->nodesetval is empty
Back in directory_empty, we get one of two responses.
1. Before fix (1) mentioned above, we get result -1 on Access Denied from AWS S3. This is because the david_stuff path is missing the / prefix, which was noticed by viewing the CURL commands to AWS S3. Without the / prefix, the constrained IAM permissions cause the access denied to get returned by AWS S3
2. After fix (1) mentioned above and before fix (2), we get result -39 which responds to -ENOTEMPTY
  - -ENOTEMPTY in this case means that, although we got the expression evaluation trying to append objects from xml. head, which is of type S3ObjList, still has objects under it. Which means that there are in fact objects under the path /david_stuff.
However, even though we know there are objects under the path, we return this warning and do not display the s3 prefix david_stuff
- After fix (2) mentioned above, we instead handle -ENOTEMPTY case and now display the s3 prefix david_stuff as a directory

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/2399 **Author:** [@jmcarpenter2](https://github.com/jmcarpenter2) **Created:** 1/23/2024 **Status:** ✅ Merged **Merged:** 2/3/2024 **Merged by:** [@ggtakec](https://github.com/ggtakec) **Base:** `master` ← **Head:** `jmc/fix-list-bucket` --- ### 📝 Commits (1) - [`bf85b38`](https://github.com/s3fs-fuse/s3fs-fuse/commit/bf85b384150f7903271e4131cbd26391a5c1455e) fix: ListBucket edge cases ### 📊 Changes **1 file changed** (+2 additions, -3 deletions) <details> <summary>View changed files</summary> 📝 `src/s3fs.cpp` (+2 -3) </details> ### 📄 Description  This pull request resolves an edge case when the XML fails to add objects to the ListBucket response, and correctly identifies that there are still objects under the `path` provided. This can be caused by slightly restricted (but still correct) IAM permissions, where you add a condition on the s3:prefix to the s3:ListBucket permissions. Specifically, this issue only occurs in special cases where there are special characters in the prefix/object path. ### Relevant Issue (if applicable)  https://github.com/s3fs-fuse/s3fs-fuse/issues/2129#issuecomment-1889763779 ### Details  There are two changes in this pull request. 1. We apply a prefix to the `reiter` string before checking if `reiter` is an empty directory. This resolves an `Access Denied` response from S3, because for some reason the [full path did not include the `/` separator between the base prefix and the rest of the path.](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs_util.cpp#L55-L56) 2. We handle the case where `directory_empty` function returns `-ENOTEMPTY`, which implies that while adding objects from the XML response of the ListBucket CURL command, [it failed but there are still objects under the path provided](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs.cpp#L1333-L1335). This failure mode appears to happen with the special IAM policy condition mentioned above in combination with special characters in the object or prefix path. By handling this special case we getStatCacheData from the updated dirpath instead and proceed with the s3 prefix appearing in the mount. ### Demonstration of fix **Basic setup** Version of s3fs being used (s3fs --version) `1.93` Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse) `2.9.9` Kernel information (uname -r) `4.14.334-252.552.amzn2.x86_64` GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` Toy example, based on real issue How to run s3fs, if applicable ``` s3fs "david-bucket:/home/David" "/home/jovyan/work/personal" -o sigv4,iam_role=auto -o endpoint=us-east-2 -o url=https://s3.us-east-2.amazonaws.com \ -o allow_other -o uid=1000,gid=1000 -o umask="0000" \ -o compat_dir,complement_stat,nonempty -o dbglevel=warn -f & ``` s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` 2024-01-12T18:14:02.304Z [ERR] s3fs.cpp:list_bucket(3402): ListBucketRequest returns with error. 2024-01-12T18:14:02.304Z [ERR] s3fs.cpp:directory_empty(1259): list_bucket returns error. 2024-01-12T18:14:02.304Z [WAN] s3fs.cpp:readdir_multi_head(3299): david_stuff/ object does not have any object under it(errno=-1), 2024-01-12T18:14:02.309Z [ERR] curl.cpp:RequestPerform(2566): HTTP response code 403, returning EPERM. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>XXX</RequestId><HostId>XXX</HostId></Error> ``` **IAM Policy** ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "bucket0nav", "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::david-bucket" ], "Condition": { "StringEquals": { "s3:prefix": [ "", "home", "home/David" ], "s3:delimiter": [ "/" ] } } }, { "Sid": "bucket0recurse", "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::david-bucket" ], "Condition": { "StringLike": { "s3:prefix": [ "home/David/*" ] } } }, { "Sid": "writeaccess", "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:GetObjectVersion", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Effect": "Allow", "Resource": [ "arn:aws:s3:::david-bucket/home/David/*", ] } ] } ``` We are trying to list bucket which has contents like the following ``` home |------David |------david_stuff |-------Life and Health History Survey-responses.json |-------file_2.txt |-------etc.. ``` **Details** It appears that the spaces in the object key Life and Health History Survey-responses.json under the david_stuff prefix is causing issues somehow in relationship with the listbucket permissions on home/David/* To demonstrate how this fixes the issue, elaborating here on the failure mode workflow, given the example setup: - Mount mountpoint `david-bucket:/home/David` to `/home/jovyan/work/personal`, with prefix under it `david_stuff/` - function [`readdir_multi_head`](https://github.com/manifoldai/s3fs-fuse/blob/master/src/s3fs.cpp#L3371) invoked on mount prefix `/home/David` - function [`directory_empty`](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs.cpp#L1324) invoked on path `david_stuff` - Inside of function [`list_bucket`](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs.cpp#L3452), [`append_objects_from_xml` is invoked](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs.cpp#L3526C17-L3526C40) - [`append_objects_from_xml_xe`](https://github.com/manifoldai/s3fs-fuse/blob/master/src/s3fs_xml.cpp#L431-L432) is invoked, trying to parse the Contents/Key or the CommonPrefixes/Prefix - here we get [`contents_xp ->nodesetval is empty`](https://github.com/manifoldai/s3fs-fuse/blob/master/src/s3fs_xml.cpp#L329) - Back in [`directory_empty`](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs.cpp#L1329-L1331), we get one of two responses. 1) Before fix (1) mentioned above, we get result `-1` on `Access Denied` from AWS S3. This is because the `david_stuff` path is missing the `/` prefix, which was noticed by viewing the CURL commands to AWS S3. **Without the `/` prefix, the constrained IAM permissions cause the access denied to get returned by AWS S3** 2) After fix (1) mentioned above and before fix (2), we get result `-39` which responds to [`-ENOTEMPTY`](https://github.com/manifoldai/s3fs-fuse/blob/jmc/fix-list-bucket/src/s3fs.cpp#L1333-L1335) - `-ENOTEMPTY` in this case means that, although we got the expression evaluation trying to append objects from xml. `head`, which is of type `S3ObjList`, still has objects under it. Which means that there are in fact objects under the path `/david_stuff`. - However, even though we know there are objects under the path, we return [this warning and do not display the s3 prefix `david_stuff`](https://github.com/manifoldai/s3fs-fuse/blob/master/src/s3fs.cpp#L3394-L3396) - After fix (2) mentioned above, **we instead handle `-ENOTEMPTY` case and now display the s3 prefix `david_stuff` as a directory** --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>