[GH-ISSUE #2300] s3fs returns I/O errors when use_cache is used #1145

New issue

Open

opened 2026-03-04 01:51:43 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 01:51:43 +03:00

Owner

Originally created by @eric-seppanen on GitHub (Aug 31, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2300

I tried using the use_cache option, but I get errors when trying to read a file:

$ s3fs my-bucket s3 -o use_cache=./s3cache
$ less s3/logs/2023-08-30/some_file
s3/logs/2023-08-30/some_file: Input/output error
$ umount s3

but the same bucket works fine without use_cache:

$ s3fs my-bucket s3
$ less s3/logs/2023-08-30/some_file
"s3/logs/2023-08-30/some_file" may be a binary file.  See it anyway? (CTRL-C)
$ umount s3

(bucket and paths redacted, apologies).

If I capture info logs I see a lot of strange errors where s3fs seems to be searching for strange paths in s3 (note the trailing ~):

2023-08-30T21:11:20.347Z [INF] s3fs.cpp:s3fs_getattr(778): [path=/logs/2023-08-30/~]
2023-08-30T21:11:20.347Z [INF]       curl.cpp:HeadRequest(3044): [tpath=/logs/2023-08-30/~]
2023-08-30T21:11:20.347Z [INF]       curl.cpp:PreHeadRequest(3004): [tpath=/logs/2023-08-30/~][bpath=][save=][sseckeypos=18446744073709551615]
2023-08-30T21:11:20.347Z [INF]       curl_util.cpp:prepare_url(254): URL is https://s3.amazonaws.com/my-bucket/logs/2023-08-30/~
2023-08-30T21:11:20.347Z [INF]       curl_util.cpp:prepare_url(287): URL changed is https://my-bucket.s3.amazonaws.com/logs/2023-08-30/~
2023-08-30T21:11:20.347Z [INF]       curl.cpp:insertV4Headers(2680): computing signature [HEAD] [/logs/2023-08-30/~] [] []
2023-08-30T21:11:20.347Z [INF]       curl_util.cpp:url_to_host(331): url is https://s3.amazonaws.com
2023-08-30T21:11:20.389Z [INF]       curl.cpp:RequestPerform(2368): HTTP response code 404 was returned, returning ENOENT
2023-08-30T21:11:20.389Z [INF]       curl.cpp:HeadRequest(3044): [tpath=/logs/2023-08-30/~/]
2023-08-30T21:11:20.389Z [INF]       curl.cpp:PreHeadRequest(3004): [tpath=/logs/2023-08-30/~/][bpath=][save=][sseckeypos=18446744073709551615]
2023-08-30T21:11:20.389Z [INF]       curl_util.cpp:prepare_url(254): URL is https://s3.amazonaws.com/my-bucket/logs/2023-08-30/~/
2023-08-30T21:11:20.389Z [INF]       curl_util.cpp:prepare_url(287): URL changed is https://my-bucket.s3.amazonaws.com/logs/2023-08-30/~/
2023-08-30T21:11:20.389Z [INF]       curl.cpp:insertV4Headers(2680): computing signature [HEAD] [/logs/2023-08-30/~/] [] []
2023-08-30T21:11:20.389Z [INF]       curl_util.cpp:url_to_host(331): url is https://s3.amazonaws.com
2023-08-30T21:11:20.432Z [INF]       curl.cpp:RequestPerform(2368): HTTP response code 404 was returned, returning ENOENT
2023-08-30T21:11:20.432Z [INF]       curl.cpp:HeadRequest(3044): [tpath=/logs/2023-08-30/~_$folder$]
2023-08-30T21:11:20.432Z [INF]       curl.cpp:PreHeadRequest(3004): [tpath=/logs/2023-08-30/~_$folder$][bpath=][save=][sseckeypos=18446744073709551615]
2023-08-30T21:11:20.432Z [INF]       curl_util.cpp:prepare_url(254): URL is https://s3.amazonaws.com/my-bucket/logs/2023-08-30/~_%24folder%24
2023-08-30T21:11:20.432Z [INF]       curl_util.cpp:prepare_url(287): URL changed is https://my-bucket.s3.amazonaws.com/logs/2023-08-30/~_%24folder%24
2023-08-30T21:11:20.432Z [INF]       curl.cpp:insertV4Headers(2680): computing signature [HEAD] [/logs/2023-08-30/~_$folder$] [] []
2023-08-30T21:11:20.432Z [INF]       curl_util.cpp:url_to_host(331): url is https://s3.amazonaws.com
2023-08-30T21:11:20.474Z [INF]       curl.cpp:RequestPerform(2368): HTTP response code 404 was returned, returning ENOENT

This looks very strange to me. Am I using use_cache incorrectly? The cache directory is just an empty directory next to my mountpoint.

Additional Information

Version of s3fs being used (`s3fs --version`)

(Ubuntu version 1.90-1, on Ubuntu 22.04)
$ s3fs --version
Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt)

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

Stock packages from Ubuntu 22.04:

$ dpkg -l |grep fuse
ii  fuse3                                      3.10.5-1build1                            amd64        Filesystem in Userspace (3.x version)
ii  gvfs-fuse                                  1.48.2-0ubuntu1                           amd64        userspace virtual filesystem - fuse server
ii  libfuse2:amd64                             2.9.9-5ubuntu3                            amd64        Filesystem in Userspace (library)
ii  libfuse3-3:amd64                           3.10.5-1build1                            amd64        Filesystem in Userspace (library) (3.x version)

Kernel information (`uname -r`)

$ uname -r
6.2.0-26-generic

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

How to run s3fs, if applicable

[x] command line
[] /etc/fstab

s3fs my-bucket s3 -o use_cache=./s3cache

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

Aug 30 18:29:48 ericwork s3fs[2186069]: s3fs version 1.90(unknown) : s3fs -o use_cache=./s3cache my-bucket s3
Aug 30 18:29:48 ericwork s3fs[2186069]: Loaded mime information from /etc/mime.types
Aug 30 18:29:48 ericwork s3fs[2186072]: init v1.90(commit:unknown) with GnuTLS(gcrypt)

Originally created by @eric-seppanen on GitHub (Aug 31, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2300 I tried using the `use_cache` option, but I get errors when trying to read a file: ``` $ s3fs my-bucket s3 -o use_cache=./s3cache $ less s3/logs/2023-08-30/some_file s3/logs/2023-08-30/some_file: Input/output error $ umount s3 ``` but the same bucket works fine without `use_cache`: ``` $ s3fs my-bucket s3 $ less s3/logs/2023-08-30/some_file "s3/logs/2023-08-30/some_file" may be a binary file. See it anyway? (CTRL-C) $ umount s3 ``` (bucket and paths redacted, apologies). If I capture info logs I see a lot of strange errors where s3fs seems to be searching for strange paths in s3 (note the trailing `~`): ``` 2023-08-30T21:11:20.347Z [INF] s3fs.cpp:s3fs_getattr(778): [path=/logs/2023-08-30/~] 2023-08-30T21:11:20.347Z [INF] curl.cpp:HeadRequest(3044): [tpath=/logs/2023-08-30/~] 2023-08-30T21:11:20.347Z [INF] curl.cpp:PreHeadRequest(3004): [tpath=/logs/2023-08-30/~][bpath=][save=][sseckeypos=18446744073709551615] 2023-08-30T21:11:20.347Z [INF] curl_util.cpp:prepare_url(254): URL is https://s3.amazonaws.com/my-bucket/logs/2023-08-30/~ 2023-08-30T21:11:20.347Z [INF] curl_util.cpp:prepare_url(287): URL changed is https://my-bucket.s3.amazonaws.com/logs/2023-08-30/~ 2023-08-30T21:11:20.347Z [INF] curl.cpp:insertV4Headers(2680): computing signature [HEAD] [/logs/2023-08-30/~] [] [] 2023-08-30T21:11:20.347Z [INF] curl_util.cpp:url_to_host(331): url is https://s3.amazonaws.com 2023-08-30T21:11:20.389Z [INF] curl.cpp:RequestPerform(2368): HTTP response code 404 was returned, returning ENOENT 2023-08-30T21:11:20.389Z [INF] curl.cpp:HeadRequest(3044): [tpath=/logs/2023-08-30/~/] 2023-08-30T21:11:20.389Z [INF] curl.cpp:PreHeadRequest(3004): [tpath=/logs/2023-08-30/~/][bpath=][save=][sseckeypos=18446744073709551615] 2023-08-30T21:11:20.389Z [INF] curl_util.cpp:prepare_url(254): URL is https://s3.amazonaws.com/my-bucket/logs/2023-08-30/~/ 2023-08-30T21:11:20.389Z [INF] curl_util.cpp:prepare_url(287): URL changed is https://my-bucket.s3.amazonaws.com/logs/2023-08-30/~/ 2023-08-30T21:11:20.389Z [INF] curl.cpp:insertV4Headers(2680): computing signature [HEAD] [/logs/2023-08-30/~/] [] [] 2023-08-30T21:11:20.389Z [INF] curl_util.cpp:url_to_host(331): url is https://s3.amazonaws.com 2023-08-30T21:11:20.432Z [INF] curl.cpp:RequestPerform(2368): HTTP response code 404 was returned, returning ENOENT 2023-08-30T21:11:20.432Z [INF] curl.cpp:HeadRequest(3044): [tpath=/logs/2023-08-30/~_$folder$] 2023-08-30T21:11:20.432Z [INF] curl.cpp:PreHeadRequest(3004): [tpath=/logs/2023-08-30/~_$folder$][bpath=][save=][sseckeypos=18446744073709551615] 2023-08-30T21:11:20.432Z [INF] curl_util.cpp:prepare_url(254): URL is https://s3.amazonaws.com/my-bucket/logs/2023-08-30/~_%24folder%24 2023-08-30T21:11:20.432Z [INF] curl_util.cpp:prepare_url(287): URL changed is https://my-bucket.s3.amazonaws.com/logs/2023-08-30/~_%24folder%24 2023-08-30T21:11:20.432Z [INF] curl.cpp:insertV4Headers(2680): computing signature [HEAD] [/logs/2023-08-30/~_$folder$] [] [] 2023-08-30T21:11:20.432Z [INF] curl_util.cpp:url_to_host(331): url is https://s3.amazonaws.com 2023-08-30T21:11:20.474Z [INF] curl.cpp:RequestPerform(2368): HTTP response code 404 was returned, returning ENOENT ``` This looks very strange to me. Am I using `use_cache` incorrectly? The cache directory is just an empty directory next to my mountpoint. ### Additional Information #### Version of s3fs being used (`s3fs --version`) ``` (Ubuntu version 1.90-1, on Ubuntu 22.04) $ s3fs --version Amazon Simple Storage Service File System V1.90 (commit:unknown) with GnuTLS(gcrypt) ``` #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) Stock packages from Ubuntu 22.04: ``` $ dpkg -l |grep fuse ii fuse3 3.10.5-1build1 amd64 Filesystem in Userspace (3.x version) ii gvfs-fuse 1.48.2-0ubuntu1 amd64 userspace virtual filesystem - fuse server ii libfuse2:amd64 2.9.9-5ubuntu3 amd64 Filesystem in Userspace (library) ii libfuse3-3:amd64 3.10.5-1build1 amd64 Filesystem in Userspace (library) (3.x version) ``` #### Kernel information (`uname -r`) ``` $ uname -r 6.2.0-26-generic ``` #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) ``` cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` #### How to run s3fs, if applicable  [x] command line [] /etc/fstab  ``` s3fs my-bucket s3 -o use_cache=./s3cache ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` Aug 30 18:29:48 ericwork s3fs[2186069]: s3fs version 1.90(unknown) : s3fs -o use_cache=./s3cache my-bucket s3 Aug 30 18:29:48 ericwork s3fs[2186069]: Loaded mime information from /etc/mime.types Aug 30 18:29:48 ericwork s3fs[2186072]: init v1.90(commit:unknown) with GnuTLS(gcrypt) ```