[GH-ISSUE #1973] Some Nested Directories Are Inaccessible

kerem commented

2026-03-04 01:50:31 +03:00

Owner

Originally created by @evanlalo on GitHub (Jun 29, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1973

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

1.91

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.4

Kernel information (uname -r)

4.14.200-116.320.amzn1.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

s3fs command line used, if applicable

s3fs {bucket} {path} -o passwd_file=/etc/passwd-s3fs
s3fs unmount {path}

Details about issue

Running into a couple of issues.

For starters, we had a recent security issue which resulted in our account being temporarily blocked. We had unauthorized EC2 usage. Our account was blocked but we went through the remediation steps AWS support asked for and our account in now unblocked. Immediately after getting unblocked, we started experiencing issues. I'm not sure if this is related but the timing is suspicious. I have verified that all necessary roles and permissions are configured as needed.

Issue 1

Our mounted bucket shows up with the all of the directories and files as expected, but a few of them say no such file or directory.

ls: cannot access default_images: No such file or directory
ls: cannot access backgrounds: No such file or directory
ls: cannot access styles: No such file or directory

The directories definitely exist in the bucket and they are listed in the mounted directory, I just cannot access them in the EC2.

Issue 2

In an effort to troubleshoot, I tried to unmount the bucket. I ran
s3fs unmount {path}

I didn't get any errors but the mount path still shows the mounted bucket and I am able to add files to the bucket from the EC2 with basic commands like touch test.txt.

My assumption would be that after running unmount I would not be able to see the bucket contents.

S3FS has been working perfectly for over a year and we would love to be able to keep using it. Any advise would be greatly appreciated.

Thank you,

Originally created by @evanlalo on GitHub (Jun 29, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1973 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) `1.91` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) `2.9.4` #### Kernel information (uname -r) `4.14.200-116.320.amzn1.x86_64` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` NAME="Amazon Linux AMI" VERSION="2018.03" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2018.03" PRETTY_NAME="Amazon Linux AMI 2018.03" ANSI_COLOR="0;33" CPE_NAME="cpe:/o:amazon:linux:2018.03:ga" HOME_URL="http://aws.amazon.com/amazon-linux-ami/" ``` #### s3fs command line used, if applicable `s3fs {bucket} {path} -o passwd_file=/etc/passwd-s3fs` `s3fs unmount {path}` ### Details about issue Running into a couple of issues. For starters, we had a recent security issue which resulted in our account being temporarily blocked. We had unauthorized EC2 usage. Our account was blocked but we went through the remediation steps AWS support asked for and our account in now unblocked. Immediately after getting unblocked, we started experiencing issues. I'm not sure if this is related but the timing is suspicious. I have verified that all necessary roles and permissions are configured as needed. ## Issue 1 Our mounted bucket shows up with the all of the directories and files as expected, but a few of them say `no such file or directory`. ``` ls: cannot access default_images: No such file or directory ls: cannot access backgrounds: No such file or directory ls: cannot access styles: No such file or directory ``` The directories definitely exist in the bucket and they are listed in the mounted directory, I just cannot access them in the EC2. ## Issue 2 In an effort to troubleshoot, I tried to unmount the bucket. I ran `s3fs unmount {path}` I didn't get any errors but the mount path still shows the mounted bucket and I am able to add files to the bucket from the EC2 with basic commands like `touch test.txt`. My assumption would be that after running `unmount` I would not be able to see the bucket contents. --- S3FS has been working perfectly for over a year and we would love to be able to keep using it. Any advise would be greatly appreciated. Thank you,

kerem closed this issue

2026-03-04 01:50:32 +03:00

kerem commented

2026-03-04 01:50:32 +03:00

Author

Owner

@evanlalo commented on GitHub (Jun 29, 2022):

Update

With not having much to loose at this point, I tried recreating one of the inaccessible directories in the mounted path.

For example, in my bucket, I had a directory called styles. This was one of the directories that s3fs said it could not access

ls: cannot access styles: No such file or directory

So I ran mkdir styles and it seemed to relink the existing directory from the bucket meaning that I could now go into that directory and see the contents and verify that it matched what was in the bucket.

The only issue is that all of the directories inside were also listed as no such file or directory.

@evanlalo commented on GitHub (Jun 29, 2022): ## Update With not having much to loose at this point, I tried recreating one of the inaccessible directories in the mounted path. For example, in my bucket, I had a directory called `styles`. This was one of the directories that s3fs said it could not access `ls: cannot access styles: No such file or directory` So I ran `mkdir styles` and it seemed to relink the existing directory from the bucket meaning that I could now go into that directory and see the contents and verify that it matched what was in the bucket. The only issue is that all of the directories inside were also listed as `no such file or directory`.

kerem commented

2026-03-04 01:50:32 +03:00

Author

Owner

@ggtakec commented on GitHub (Jun 29, 2022):

@evanlalo
I think this may be the case where the directory object(<dirname>/) does not exist and some file objects exist under that directory path.
(S3fs needs an object called <dirname>/ to recognize it as a directory, but this problem can occur because the object doesn't exist.)

In this case, try starting s3fs with the compat_dir option (if you are using the current master branch).
If you are using up to v1.91, this function should be working by default, so I think you can not identify the cause without investigating logs etc.
(Until v1.91, there is an option notsup_compat_dir to not check the case where the directory object does not exist, contrary to the current master)

If you can see the directory on s3fs, you will be able to create a directory object by carefully manipulating permissions etc. with the touch command, chmod/chown, etc.
Once the directory object is created, I don't think this issue will occur anymore.
(Alternatively, you may be able to avoid it by creating an empty object called <dirname>/ from the S3 console.)

First of all, please check if you can refer to it without operating the directory.

@ggtakec commented on GitHub (Jun 29, 2022): @evanlalo I think this may be the case where the directory object(`<dirname>/`) does not exist and some file objects exist under that directory path. (S3fs needs an object called `<dirname>/` to recognize it as a directory, but this problem can occur because the object doesn't exist.) In this case, try starting s3fs with the `compat_dir` option (if you are using the current `master branch`). If you are using up to v1.91, this function should be working by default, so I think you can not identify the cause without investigating logs etc. (Until v1.91, there is an option `notsup_compat_dir` to not check the case where the directory object does not exist, contrary to the current master) If you can see the directory on s3fs, you will be able to create a directory object by carefully manipulating permissions etc. with the touch command, chmod/chown, etc. Once the directory object is created, I don't think this issue will occur anymore. (Alternatively, you may be able to avoid it by creating an empty object called `<dirname>/` from the S3 console.) First of all, please check if you can refer to it without operating the directory.

kerem commented

2026-03-04 01:50:32 +03:00

Author

Owner

@evanlalo commented on GitHub (Jun 29, 2022):

@ggtakec thanks for your reply but I'm not sure I understand your instructions.

Here are the contents of my mounted directory.

drwxrwxr-x 1 webapp webapp      0 Jun 29 14:26 styles
?????????? ? ?      ?           ?            ? vehicles
drwxr-x--- 1 webapp webapp      0 Jun 29 16:17 vehicles_test
drwxrwxr-x 1 webapp webapp      0 Feb 23 10:33 videos

The vehicles directory exists in S3 and is filled with literally millions of records.

If I try to perform any command on the directory (cd, chown, chmod, etc..), I get an error cannot access vehicles: No such file or directory.

Just out of curiosity, I created vehicles_test in the s3 console and then copied a sub directory from the vehicles directory.

I can see the vehicles_test directory in my EC2 and I can navigate to the copied sub directory. However, that copied directory also has directories and I get the same error cannot access {directory name}: No such file or directory.

@evanlalo commented on GitHub (Jun 29, 2022): @ggtakec thanks for your reply but I'm not sure I understand your instructions. Here are the contents of my mounted directory. ``` drwxrwxr-x 1 webapp webapp 0 Jun 29 14:26 styles ?????????? ? ? ? ? ? vehicles drwxr-x--- 1 webapp webapp 0 Jun 29 16:17 vehicles_test drwxrwxr-x 1 webapp webapp 0 Feb 23 10:33 videos ``` The `vehicles` directory exists in S3 and is filled with literally millions of records. If I try to perform any command on the directory (cd, chown, chmod, etc..), I get an error `cannot access vehicles: No such file or directory`. Just out of curiosity, I created `vehicles_test` in the s3 console and then copied a sub directory from the `vehicles` directory. I can see the `vehicles_test` directory in my EC2 and I can navigate to the copied sub directory. However, that copied directory also has directories and I get the same error `cannot access {directory name}: No such file or directory`.

kerem commented

2026-03-04 01:50:33 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 3, 2022):

@evanlalo It looks like the operation method is a little different.
First, I explain about what seems to be the cause of this problem.

s3fs background operation about directories

s3fs sets attributes(http headers : x-amz-mod/gid/uid/*time) for S3 objects in order to treat them as files and directories.
However, directories are treated specially.

For example, if you create a <bucket>/<dir>/<file> object with awscli or S3 console, the only <dir>/<file> object is created on S3.
The <dir> object does not exist.

In order for s3fs to behave like a file system, directories also need attributes.
s3fs creates the directory object with attribute if user create files(directory) through s3fs.
In the above example, s3fs will create an object called <dir>/. (Note that there is a / at the end)
This <dir>/ object is 0 bytes and only the attributes are set.

I think your this issue is most likely due to the absence of the <dir>/ object.

How to check if directory object is existed(from s3fs)

s3fs has features and options that allow you to access the above <dir>/ even if it doesn't exist.
Check if your problem is due to the absence of this <dir>/ object as follows:

Check the directory for inaccessibility

If your s3fs is v1.91 or lower, start it with notsup_compat_dir.
If you're using s3fs built from the code in current master branch, you don't need to add any options.

The result of the ls command is displayed as follows:

$ ls -la
ls: cannot access no_folder: No such file or directory
ls: cannot access no_info: No such file or directory
total 76818
drwxrwxrwx 1 ggtakec users           0 Jun 21 12:18 .
drwxr-xr-x 6 root     root           71 Jun 21 12:16 ..
?????????? ? ?        ?               ?            ? no_folder
?????????? ? ?        ?               ?            ? no_info

The directories(likely) no_folder and no_info are visible, but the permissions and other attributes are unknown.
This result looks the same as your problem right now.

How to make it accessible

If your s3fs is v1.91 or lower, you don't need to add any options.
If you are using s3fs built from the code in your current master branch, start it with the compat_dir option.

The result of the ls command is displayed as follows:

$ ls -la
total 76819
drwxrwxrwx 1 ggtakec users           0 Jan  1  1970 .
drwxr-xr-x 6 root     root           71 Jun 21 12:16 ..
drwxr-x--- 1 ggtakec users           0 Jan  1  1970 no_folder
drwxr-x--- 1 ggtakec users           0 Jan  1  1970 no_info

In this way, you can see the directories, and you can also refer to the files under those directories.

What you can do (if the cause is the same as above)

By adding(not adding) compat_dir(notsup_compat_dir) as in the above example, you can refer to it even if the directory object does not exist.
Please check the version of s3fs and check the above exmaples.

Permanent response

If the cause is that the <dir>/ object doesn't exist, you can make it create a <dir>/ object for a permanent response.

First, start s3fs so that you can browse directories that are currently unavailable.(by adding(or not adding) (notsup_)compat_diroption)
Then update the attributes for that directory with commands such as chmod/chown/chgrp/touch(for update time)/etc.
With this update, s3fs will create a <dir>/ object for that directory.
And next time to run s3fs, you will be able to browse the directory without(or by) giving the option.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Jul 3, 2022): @evanlalo It looks like the operation method is a little different. First, I explain about what seems to be the cause of this problem. ### s3fs background operation about directories s3fs sets attributes(http headers : x-amz-mod/gid/uid/*time) for S3 objects in order to treat them as files and directories. However, directories are treated specially. For example, if you create a `<bucket>/<dir>/<file>` object with awscli or S3 console, the only `<dir>/<file>` object is created on S3. The `<dir>` object does not exist. In order for s3fs to behave like a file system, directories also need attributes. s3fs creates the directory object with attribute if user create files(directory) through s3fs. In the above example, s3fs will create an object called `<dir>/`. (Note that there is a `/` at the end) This `<dir>/` object is 0 bytes and only the attributes are set. I think your this issue is most likely due to the absence of the `<dir>/` object. ### How to check if directory object is existed(from s3fs) s3fs has features and options that allow you to access the above `<dir>/` even if it doesn't exist. Check if your problem is due to the absence of this `<dir>/` object as follows: #### Check the directory for inaccessibility If your s3fs is v1.91 or lower, start it with `notsup_compat_dir`. If you're using s3fs built from the code in current master branch, you don't need to add any options. The result of the ls command is displayed as follows: ``` $ ls -la ls: cannot access no_folder: No such file or directory ls: cannot access no_info: No such file or directory total 76818 drwxrwxrwx 1 ggtakec users 0 Jun 21 12:18 . drwxr-xr-x 6 root root 71 Jun 21 12:16 .. ?????????? ? ? ? ? ? no_folder ?????????? ? ? ? ? ? no_info ``` The directories(likely) `no_folder` and `no_info` are visible, but the permissions and other attributes are unknown. This result looks the same as your problem right now. #### How to make it accessible If your s3fs is v1.91 or lower, you don't need to add any options. If you are using s3fs built from the code in your current master branch, start it with the `compat_dir` option. The result of the ls command is displayed as follows: ``` $ ls -la total 76819 drwxrwxrwx 1 ggtakec users 0 Jan 1 1970 . drwxr-xr-x 6 root root 71 Jun 21 12:16 .. drwxr-x--- 1 ggtakec users 0 Jan 1 1970 no_folder drwxr-x--- 1 ggtakec users 0 Jan 1 1970 no_info ``` In this way, you can see the directories, and you can also refer to the files under those directories. ### What you can do (if the cause is the same as above) By adding(not adding) `compat_dir`(`notsup_compat_dir`) as in the above example, you can refer to it even if the directory object does not exist. Please check the version of s3fs and check the above exmaples. ### Permanent response If the cause is that the `<dir>/` object doesn't exist, you can make it create a `<dir>/` object for a permanent response. - First, start s3fs so that you can browse directories that are currently unavailable.(by adding(or not adding) `(notsup_)compat_dir`option) - Then update the attributes for that directory with commands such as `chmod`/`chown`/`chgrp`/`touch`(for update time)/etc. - With this update, s3fs will create a `<dir>/` object for that directory. - And next time to run s3fs, you will be able to browse the directory without(or by) giving the option. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:50:33 +03:00

Author

Owner

@pedhenrique commented on GitHub (Jul 12, 2022):

@ggtakec This worked for me. Thanks

@pedhenrique commented on GitHub (Jul 12, 2022): @ggtakec This worked for me. Thanks

kerem commented

2026-03-04 01:50:33 +03:00

Author

Owner

@gaul commented on GitHub (Jul 16, 2022):

Does #1986 fix this?

@gaul commented on GitHub (Jul 16, 2022): Does #1986 fix this?

kerem commented

2026-03-04 01:50:33 +03:00

Author

Owner

@gaul commented on GitHub (Jul 27, 2022):

Please reopen if symptoms persist.

@gaul commented on GitHub (Jul 27, 2022): Please reopen if symptoms persist.

kerem referenced this issue

2026-03-04 02:01:58 +03:00

[PR #997] [MERGED] Prefer HTTPS links in documentation where possible #1749

Rows
Columns

[GH-ISSUE #1973] Some Nested Directories Are Inaccessible #997

Additional Information

Version of s3fs being used (s3fs --version)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Kernel information (uname -r)

GNU/Linux Distribution, if applicable (cat /etc/os-release)

s3fs command line used, if applicable

Details about issue

Issue 1

Issue 2

Update

s3fs background operation about directories

How to check if directory object is existed(from s3fs)

Check the directory for inaccessibility

How to make it accessible

What you can do (if the cause is the same as above)

Permanent response