starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #2067] ls, permission denied for filenames containing CR (0x13) character #1042

New issue

Closed

opened 2026-03-04 01:50:54 +03:00 by kerem · 7 comments

kerem commented

2026-03-04 01:50:54 +03:00

Owner

Originally created by @jalmasi on GitHub (Nov 25, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2067

Did you know that linux file names can contain CRLF (0x13) character? :>
I guess everybody knows about special characters, but 0x13 is even more special. If you manage to create a file containing 0x13, you're not going to see it in the file name. You may see it as <0x13>, ^S or DC3 in editors/viewers if you redirect ls to file, but linux console will just not render it.

Then you mount a bucket with s3fs, copy that file over to a directory there, issue ls, and get permission denied. Everything works, except that one directory containing the evil file(s).
But you can list the directory content using awscli, permission denied will not be there. And neither will 0x13, because it's so very special.
S3 UI for the bucket also does not display the character, and does not allow file removal. Yet it does allow to remove the character, everything works afterwards. You just need to know where it is. Only just :)

This isn't exactly a bug report. I don't expect anybody to do anything about this, I'm just trying to save time and nerves of the next guy that runs into it.
The above provides enough info to diagnose and reproduce the issue. Fix is obvious, renaming the file.
Hard part is matching 0x13. I had a small java program written for that, but language doesn't matter. Couldn't use regex, because, like, what's the escape sequence? :) I guess utf escape sequences might do.
Well, HTH :>

Originally created by @jalmasi on GitHub (Nov 25, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2067 Did you know that linux file names can contain CRLF (0x13) character? :> I guess everybody knows about special characters, but 0x13 is even more special. If you manage to create a file containing 0x13, you're not going to see it in the file name. You may see it as <0x13>, ^S or DC3 in editors/viewers if you redirect ls to file, but linux console will just not render it. Then you mount a bucket with s3fs, copy that file over to a directory there, issue ls, and get permission denied. Everything works, except that one directory containing the evil file(s). But you can list the directory content using awscli, permission denied will not be there. And neither will 0x13, because it's so very special. S3 UI for the bucket also does not display the character, and does not allow file removal. Yet it does allow to remove the character, everything works afterwards. You just need to know where it is. Only just :) This isn't exactly a bug report. I don't expect anybody to do anything about this, I'm just trying to save time and nerves of the next guy that runs into it. The above provides enough info to diagnose and reproduce the issue. Fix is obvious, renaming the file. Hard part is matching 0x13. I had a small java program written for that, but language doesn't matter. Couldn't use regex, because, like, what's the escape sequence? :) I guess utf escape sequences might do. Well, HTH :>

kerem closed this issue

2026-03-04 01:50:54 +03:00

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@simonfelding commented on GitHub (Feb 1, 2023):

@jalmasi the regex way is to match 0x0D (it's character 13 in decimal, 0D in hexadecimal!). \r should do it too :)

@simonfelding commented on GitHub (Feb 1, 2023): @jalmasi the regex way is to match 0x0D (it's character 13 in decimal, 0D in hexadecimal!). \r should do it too :)

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 12, 2023):

I did a quick check.
I created objects with filenames containing \n and \r in my S3 bucket and referenced them by mounting them with s3fs.
And they were the same as the result displayed by ls in the file Linux created on the local disk.
Both are displayed as ?, such as SAMPLE?.
Inside s3fs, these special characters appear to be processed exactly as received from the S3 server.

@jalmasi
What's wrong with s3fs compared to normal Linux commands?

@ggtakec commented on GitHub (Mar 12, 2023): I did a quick check. I created objects with filenames containing `\n` and `\r` in my S3 bucket and referenced them by mounting them with s3fs. And they were the same as the result displayed by `ls` in the file Linux created on the local disk. Both are displayed as `?`, such as `SAMPLE?`. Inside s3fs, these special characters appear to be processed exactly as received from the S3 server. @jalmasi What's wrong with s3fs compared to normal Linux commands?

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@jalmasi commented on GitHub (Mar 12, 2023):

@ggtakec simple "find ." on the mounted bucket would end up with permission denied. But note that these weren't bucket objects created in the bucket, these were files on ext4 filesystem that contained special characters. Then the bucket was mounted, and files copied over.

@jalmasi commented on GitHub (Mar 12, 2023): @ggtakec simple "find ." on the mounted bucket would end up with permission denied. But note that these weren't bucket objects created in the bucket, these were files on ext4 filesystem that contained special characters. Then the bucket was mounted, and files copied over.

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 21, 2023):

@jalmasi
I found the cause of this bug and posted a PR #2136 to fix it.

The cause was that the character code was converted by libxml, CR (0x0D) characters were converted to LF(0x0A), and the file could not be detected.

If you can build this modified code, please try to check it.
Thanks in advance for your assistance.

@ggtakec commented on GitHub (Mar 21, 2023): @jalmasi I found the cause of this bug and posted a PR #2136 to fix it. The cause was that the character code was converted by libxml, `CR` (`0x0D`) characters were converted to `LF`(`0x0A`), and the file could not be detected. If you can build this modified code, please try to check it. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@jalmasi commented on GitHub (Mar 22, 2023):

@ggtakec awesome!
I can confirm it's fixed.
I don't have access to original data and systems any longer, so I reproduced as follows:

created a file with \n and another with \r in the filename
copied to s3 bucket mounted with your patched s3fs
mounted on ubuntu 20 version (s3fs v1.86)

patched find /mnt/test/files/
/mnt/test/files/
/mnt/test/files/a?test.txt
/mnt/test/files/another?test.txt
/mnt/test/files/blah.txt

patched ls -l /mnt/test/files/
total 2.5K
drwxr-xr-x 1 root root 0 Mar 22 11:11 ./
drwxr-xr-x 1 root root 0 Mar 22 11:12 ../
-rw-r--r-- 1 root root 0 Mar 22 11:10 'a'$'\n''test.txt'
-rw-r--r-- 1 root root 0 Mar 22 11:10 'another'$'\r''test.txt'
-rw-r--r-- 1 root root 0 Mar 22 11:24 blah.txt

old find /mnt/test/files/
/mnt/test/files/
/mnt/test/files/a?test.txt
find: ‘/mnt/test/files/another\ntest.txt’: No such file or directory
/mnt/test/files/another?test.txt
/mnt/test/files/blah.txt

old ls -l /mnt/test/files/
ls: cannot access '/mnt/test/files/another'$'\n''test.txt': No such file or directory
total 2.0K
drwxr-xr-x 1 root root 0 Mar 22 11:11 ./
drwxr-xr-x 1 root root 0 Mar 22 11:12 ../
-rw-r--r-- 1 root root 0 Mar 22 11:10 'a'$'\n''test.txt'
?????????? ? ? ? ? ? 'another'$'\n''test.txt'
-rw-r--r-- 1 root root 0 Mar 22 11:24 blah.txt

@jalmasi commented on GitHub (Mar 22, 2023): @ggtakec awesome! I can confirm it's fixed. I don't have access to original data and systems any longer, so I reproduced as follows: - created a file with \n and another with \r in the filename - copied to s3 bucket mounted with your patched s3fs - mounted on ubuntu 20 version (s3fs v1.86) patched find /mnt/test/files/ /mnt/test/files/ /mnt/test/files/a?test.txt /mnt/test/files/another?test.txt /mnt/test/files/blah.txt patched ls -l /mnt/test/files/ total 2.5K drwxr-xr-x 1 root root 0 Mar 22 11:11 ./ drwxr-xr-x 1 root root 0 Mar 22 11:12 ../ -rw-r--r-- 1 root root 0 Mar 22 11:10 'a'$'\n''test.txt' -rw-r--r-- 1 root root 0 Mar 22 11:10 'another'$'\r''test.txt' -rw-r--r-- 1 root root 0 Mar 22 11:24 blah.txt old find /mnt/test/files/ /mnt/test/files/ /mnt/test/files/a?test.txt find: ‘/mnt/test/files/another\ntest.txt’: No such file or directory /mnt/test/files/another?test.txt /mnt/test/files/blah.txt old ls -l /mnt/test/files/ ls: cannot access '/mnt/test/files/another'$'\n''test.txt': No such file or directory total 2.0K drwxr-xr-x 1 root root 0 Mar 22 11:11 ./ drwxr-xr-x 1 root root 0 Mar 22 11:12 ../ -rw-r--r-- 1 root root 0 Mar 22 11:10 'a'$'\n''test.txt' ?????????? ? ? ? ? ? 'another'$'\n''test.txt' -rw-r--r-- 1 root root 0 Mar 22 11:24 blah.txt

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@ggtakec commented on GitHub (Mar 26, 2023):

@jalmasi Thank you for confirming.
I merged #2136.
This issue will be closed, but if the problem still exists, please reopen or post a new issue.

@ggtakec commented on GitHub (Mar 26, 2023): @jalmasi Thank you for confirming. I merged #2136. This issue will be closed, but if the problem still exists, please reopen or post a new issue.

kerem commented

2026-03-04 01:50:55 +03:00

Author

Owner

@jwarnn commented on GitHub (Oct 20, 2023):

I think I am seeing this problem crop up again. We are trying to use s3fs to upload files from a machine to s3 to ec2. Some of the files from the machine have '#' in the name; could that still be causing an issue?

A little more info. This is a DNA sequencer running on a windows operating system, it outputs a run folder, thats being saved in a shared drive, thats going to an s3 bucket, and then thats going to an EC2 instance using s3fs.

@jwarnn commented on GitHub (Oct 20, 2023): I think I am seeing this problem crop up again. We are trying to use s3fs to upload files from a machine to s3 to ec2. Some of the files from the machine have '#' in the name; could that still be causing an issue? A little more info. This is a DNA sequencer running on a windows operating system, it outputs a run folder, thats being saved in a shared drive, thats going to an s3 bucket, and then thats going to an EC2 instance using s3fs.