starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2670] Issues with IONOS Cloud S3 "Input/Output error" #1274

New issue
Open
opened 2026-03-04 01:52:44 +03:00 by kerem · 1 comment
kerem commented 2026-03-04 01:52:44 +03:00
Owner
Copy link

Originally created by @agowa on GitHub (May 12, 2025).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2670

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.9

Provider (AWS, OVH, Hetzner, iDrive E2, ...)

IONOS Cloud

Kernel information (uname -r)

6.14.5-arch1-1

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

How to run s3fs, if applicable

I'm using a systemd mount-unit. However I'm also going to attach the resulting command line just in case.
[x] command line: s3fs 88dbcaef-04bc-4e15-a177-c760a46ca70b /mnt/s3fs -o rw,dbglevel=info,curldbg,logfile=/var/log/s3fs.log,nosuid,nodev,noexec,noatime,allow_other,uid=1000,gid=984,default_permissions,passwd_file=/home/user/.passwd-s3fs,url=https://s3.eu-central-3.ionoscloud.com,tmpdir=/mnt/s3fs-tmp,use_cache=/mnt/s3fs-cache,max_stat_cache_size=2500,parallel_count=48,ensure_diskfree=10240,host=https://s3.eu-central-3.ionoscloud.com,endpoint=eu-central-3,enable_content_md5
[] /etc/fstab
[x] systemd

# /etc/systemd/system/mnt-s3fs.mount
[Unit]
Description=S3FS
Before=remote-fs.target

[Mount]
What=88dbcaef-04bc-4e15-a177-c760a46ca70b
Where=/mnt/s3fs
Type=fuse.s3fs
Options=dbglevel=info,curldbg,logfile=/var/log/s3fs.log,_netdev,rw,nosuid,nodev,noexec,noatime,allow_other,uid=1000,gid=984,default_permissions,passwd_file=/home/user/.passwd-s3fs,url=https://s3.eu-central-3.ionoscloud.com,tmpdir=/mnt/s3fs-tmp,use_cache=/mnt/s3fs-cache,max_stat_cache_size=2500,parallel_count=48,ensure_diskfree=10240,host=https://s3.eu-central-3.ionoscloud.com,endpoint=eu-central-3,enable_content_md5

[Install]
WantedBy=remote-fs.target
WantedBy=multi-user.target

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

2025-05-12T06:53:26.234Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/]
2025-05-12T06:53:26.236Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/test12345]
2025-05-12T06:53:26.236Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/test12345]
2025-05-12T06:53:26.236Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/test12345][bpath=][save=][sseckeypos=<<<removed>>>]
2025-05-12T06:53:26.236Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345
2025-05-12T06:53:26.236Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345
2025-05-12T06:53:26.236Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345] [] []
2025-05-12T06:53:26.236Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.236Z [CURL DBG] * Connection 10 seems to be dead

2025-05-12T06:53:26.236Z [CURL DBG] * shutting down connection #10

2025-05-12T06:53:26.236Z [CURL DBG] * Hostname 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com was found in DNS cache

2025-05-12T06:53:26.236Z [CURL DBG] *   Trying [2a01:239:100:4ff::fffe]:443...

2025-05-12T06:53:26.255Z [CURL DBG] * SSL reusing session with ALPN '-'

2025-05-12T06:53:26.256Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1):

2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2):

2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):

2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Certificate (11):

2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, CERT verify (15):

2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Finished (20):

2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Finished (20):

2025-05-12T06:53:26.275Z [CURL DBG] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey

2025-05-12T06:53:26.275Z [CURL DBG] * Server certificate:

2025-05-12T06:53:26.275Z [CURL DBG] *  subject: C=DE; ST=Rheinland-Pfalz; O=IONOS SE; CN=s3.eu-central-3.ionoscloud.com

2025-05-12T06:53:26.275Z [CURL DBG] *  start date: Feb 27 00:00:00 2025 GMT

2025-05-12T06:53:26.275Z [CURL DBG] *  expire date: Feb 27 23:59:59 2026 GMT

2025-05-12T06:53:26.275Z [CURL DBG] *  subjectAltName: host "88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com" matched cert's "*.s3.eu-central-3.ionoscloud.com"

2025-05-12T06:53:26.275Z [CURL DBG] *  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Organization Validation Secure Server CA

2025-05-12T06:53:26.275Z [CURL DBG] *  SSL certificate verify ok.

2025-05-12T06:53:26.275Z [CURL DBG] *   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256

2025-05-12T06:53:26.275Z [CURL DBG] *   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384

2025-05-12T06:53:26.275Z [CURL DBG] *   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384

2025-05-12T06:53:26.275Z [CURL DBG] * Connected to 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com (2a01:239:100:4ff::fffe) port 443

2025-05-12T06:53:26.275Z [CURL DBG] * using HTTP/1.x

2025-05-12T06:53:26.275Z [CURL DBG] > HEAD /test12345 HTTP/1.1
2025-05-12T06:53:26.275Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.275Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL)
2025-05-12T06:53:26.275Z [CURL DBG] > Accept: */*
2025-05-12T06:53:26.275Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>>
2025-05-12T06:53:26.275Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2025-05-12T06:53:26.275Z [CURL DBG] > x-amz-date: 20250512T065326Z
2025-05-12T06:53:26.275Z [CURL DBG] > 
2025-05-12T06:53:26.275Z [CURL DBG] * Request completely sent off

2025-05-12T06:53:26.291Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

2025-05-12T06:53:26.291Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

2025-05-12T06:53:26.294Z [CURL DBG] < HTTP/1.1 404 Not Found
2025-05-12T06:53:26.294Z [CURL DBG] < content-length: 283
2025-05-12T06:53:26.294Z [CURL DBG] < x-amz-request-id: tx000001495420fd3f39f63-0068219ae6-11d1209-eu-central-3
2025-05-12T06:53:26.294Z [CURL DBG] < accept-ranges: bytes
2025-05-12T06:53:26.294Z [CURL DBG] < content-type: application/xml
2025-05-12T06:53:26.294Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT
2025-05-12T06:53:26.294Z [CURL DBG] < 
2025-05-12T06:53:26.294Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact

2025-05-12T06:53:26.294Z [INF]       curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2025-05-12T06:53:26.294Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/test12345/]
2025-05-12T06:53:26.294Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/test12345/][bpath=][save=][sseckeypos=<<<removed>>>]
2025-05-12T06:53:26.294Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345/
2025-05-12T06:53:26.294Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345/
2025-05-12T06:53:26.294Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345/] [] []
2025-05-12T06:53:26.294Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.294Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com

2025-05-12T06:53:26.294Z [CURL DBG] > HEAD /test12345/ HTTP/1.1
2025-05-12T06:53:26.294Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.294Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL)
2025-05-12T06:53:26.294Z [CURL DBG] > Accept: */*
2025-05-12T06:53:26.294Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>>
2025-05-12T06:53:26.294Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2025-05-12T06:53:26.294Z [CURL DBG] > x-amz-date: 20250512T065326Z
2025-05-12T06:53:26.294Z [CURL DBG] > 
2025-05-12T06:53:26.294Z [CURL DBG] * Request completely sent off

2025-05-12T06:53:26.313Z [CURL DBG] < HTTP/1.1 404 Not Found
2025-05-12T06:53:26.313Z [CURL DBG] < content-length: 283
2025-05-12T06:53:26.313Z [CURL DBG] < x-amz-request-id: tx000000e68cf4acc5f9d1c-0068219ae6-11d11f5-eu-central-3
2025-05-12T06:53:26.313Z [CURL DBG] < accept-ranges: bytes
2025-05-12T06:53:26.313Z [CURL DBG] < content-type: application/xml
2025-05-12T06:53:26.313Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT
2025-05-12T06:53:26.313Z [CURL DBG] < 
2025-05-12T06:53:26.313Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact

2025-05-12T06:53:26.313Z [INF]       curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2025-05-12T06:53:26.313Z [INF]   s3fs.cpp:list_bucket(3434): [path=/test12345]
2025-05-12T06:53:26.313Z [INF]       curl.cpp:ListBucketRequest(3749): [tpath=/test12345]
2025-05-12T06:53:26.313Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b?delimiter=/&max-keys=2&prefix=test12345/
2025-05-12T06:53:26.313Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com?delimiter=/&max-keys=2&prefix=test12345/
2025-05-12T06:53:26.313Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test12345/] []
2025-05-12T06:53:26.313Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.313Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com

2025-05-12T06:53:26.313Z [CURL DBG] > GET /?delimiter=/&max-keys=2&prefix=test12345/ HTTP/1.1
2025-05-12T06:53:26.313Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.313Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL)
2025-05-12T06:53:26.313Z [CURL DBG] > Accept: */*
2025-05-12T06:53:26.313Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>>
2025-05-12T06:53:26.313Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2025-05-12T06:53:26.313Z [CURL DBG] > x-amz-date: 20250512T065326Z
2025-05-12T06:53:26.313Z [CURL DBG] > 
2025-05-12T06:53:26.313Z [CURL DBG] * Request completely sent off

2025-05-12T06:53:26.336Z [CURL DBG] < HTTP/1.1 200 OK
2025-05-12T06:53:26.336Z [CURL DBG] < transfer-encoding: chunked
2025-05-12T06:53:26.336Z [CURL DBG] < x-amz-request-id: tx000004295d3c06aabc064-0068219ae6-11ce797-eu-central-3
2025-05-12T06:53:26.336Z [CURL DBG] < content-type: application/xml
2025-05-12T06:53:26.336Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT
2025-05-12T06:53:26.336Z [CURL DBG] < 
2025-05-12T06:53:26.336Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact

2025-05-12T06:53:26.336Z [INF]       curl.cpp:RequestPerform(2520): HTTP response code 200
2025-05-12T06:53:26.336Z [INF]       cache.cpp:AddNoObjectCache(465): add no object cache entry[path=/test12345]
2025-05-12T06:53:26.337Z [INF] s3fs.cpp:s3fs_create(1169): [path=/test12345][mode=100644][flags=0x8841]
2025-05-12T06:53:26.337Z [INF]       cache.cpp:AddStat(342): add stat cache entry[path=/test12345]
2025-05-12T06:53:26.337Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/test12345]
2025-05-12T06:53:26.337Z [ERR] fdcache_entity.cpp:Open(549): failed to open file(/mnt/s3fs-cache/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345). errno(24)
2025-05-12T06:53:26.337Z [ERR] fdcache.cpp:Open(597): failed to open and create new pseudo fd for path(/test12345).
2025-05-12T06:53:26.337Z [INF]       cache.cpp:DelStat(596): delete stat cache entry[path=/test12345]
2025-05-12T06:53:26.337Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/test12345]
2025-05-12T06:53:26.337Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/test12345]
2025-05-12T06:53:26.337Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/test12345][bpath=][save=][sseckeypos=<<<removed>>>]
2025-05-12T06:53:26.337Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345
2025-05-12T06:53:26.337Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345
2025-05-12T06:53:26.337Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345] [] []
2025-05-12T06:53:26.337Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.337Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com

2025-05-12T06:53:26.337Z [CURL DBG] > HEAD /test12345 HTTP/1.1
2025-05-12T06:53:26.337Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.338Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL)
2025-05-12T06:53:26.338Z [CURL DBG] > Accept: */*
2025-05-12T06:53:26.338Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>>
2025-05-12T06:53:26.338Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2025-05-12T06:53:26.338Z [CURL DBG] > x-amz-date: 20250512T065326Z
2025-05-12T06:53:26.338Z [CURL DBG] > 
2025-05-12T06:53:26.338Z [CURL DBG] * Request completely sent off

2025-05-12T06:53:26.355Z [CURL DBG] < HTTP/1.1 404 Not Found
2025-05-12T06:53:26.355Z [CURL DBG] < content-length: 283
2025-05-12T06:53:26.355Z [CURL DBG] < x-amz-request-id: tx000004bac52f28e4365d0-0068219ae6-11d11f5-eu-central-3
2025-05-12T06:53:26.355Z [CURL DBG] < accept-ranges: bytes
2025-05-12T06:53:26.355Z [CURL DBG] < content-type: application/xml
2025-05-12T06:53:26.355Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT
2025-05-12T06:53:26.355Z [CURL DBG] < 
2025-05-12T06:53:26.355Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact

2025-05-12T06:53:26.355Z [INF]       curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2025-05-12T06:53:26.355Z [INF]       curl.cpp:HeadRequest(3299): [tpath=/test12345/]
2025-05-12T06:53:26.355Z [INF]       curl.cpp:PreHeadRequest(3259): [tpath=/test12345/][bpath=][save=][sseckeypos=<<<removed>>>]
2025-05-12T06:53:26.355Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345/
2025-05-12T06:53:26.355Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345/
2025-05-12T06:53:26.355Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345/] [] []
2025-05-12T06:53:26.355Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.355Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com

2025-05-12T06:53:26.355Z [CURL DBG] > HEAD /test12345/ HTTP/1.1
2025-05-12T06:53:26.355Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.355Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL)
2025-05-12T06:53:26.355Z [CURL DBG] > Accept: */*
2025-05-12T06:53:26.355Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>>
2025-05-12T06:53:26.355Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2025-05-12T06:53:26.355Z [CURL DBG] > x-amz-date: 20250512T065326Z
2025-05-12T06:53:26.355Z [CURL DBG] > 
2025-05-12T06:53:26.355Z [CURL DBG] * Request completely sent off

2025-05-12T06:53:26.373Z [CURL DBG] < HTTP/1.1 404 Not Found
2025-05-12T06:53:26.373Z [CURL DBG] < content-length: 283
2025-05-12T06:53:26.373Z [CURL DBG] < x-amz-request-id: tx00000f884bddfc38c53cd-0068219ae6-11ce797-eu-central-3
2025-05-12T06:53:26.373Z [CURL DBG] < accept-ranges: bytes
2025-05-12T06:53:26.373Z [CURL DBG] < content-type: application/xml
2025-05-12T06:53:26.373Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT
2025-05-12T06:53:26.373Z [CURL DBG] < 
2025-05-12T06:53:26.373Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact

2025-05-12T06:53:26.373Z [INF]       curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2025-05-12T06:53:26.373Z [INF]   s3fs.cpp:list_bucket(3434): [path=/test12345]
2025-05-12T06:53:26.373Z [INF]       curl.cpp:ListBucketRequest(3749): [tpath=/test12345]
2025-05-12T06:53:26.373Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b?delimiter=/&max-keys=2&prefix=test12345/
2025-05-12T06:53:26.373Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com?delimiter=/&max-keys=2&prefix=test12345/
2025-05-12T06:53:26.373Z [INF]       curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test12345/] []
2025-05-12T06:53:26.373Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.373Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com

2025-05-12T06:53:26.373Z [CURL DBG] > GET /?delimiter=/&max-keys=2&prefix=test12345/ HTTP/1.1
2025-05-12T06:53:26.373Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com
2025-05-12T06:53:26.373Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL)
2025-05-12T06:53:26.373Z [CURL DBG] > Accept: */*
2025-05-12T06:53:26.373Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>>
2025-05-12T06:53:26.373Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2025-05-12T06:53:26.373Z [CURL DBG] > x-amz-date: 20250512T065326Z
2025-05-12T06:53:26.373Z [CURL DBG] > 
2025-05-12T06:53:26.373Z [CURL DBG] * Request completely sent off

2025-05-12T06:53:26.392Z [CURL DBG] < HTTP/1.1 200 OK
2025-05-12T06:53:26.392Z [CURL DBG] < transfer-encoding: chunked
2025-05-12T06:53:26.392Z [CURL DBG] < x-amz-request-id: tx000002bf0de7bf57a7436-0068219ae6-11d1209-eu-central-3
2025-05-12T06:53:26.392Z [CURL DBG] < content-type: application/xml
2025-05-12T06:53:26.392Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT
2025-05-12T06:53:26.392Z [CURL DBG] < 
2025-05-12T06:53:26.392Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact

2025-05-12T06:53:26.392Z [INF]       curl.cpp:RequestPerform(2520): HTTP response code 200
2025-05-12T06:53:26.393Z [INF]       cache.cpp:AddNoObjectCache(465): add no object cache entry[path=/test12345]

Details about issue

  1. mount s3bucket
  2. Do operations on/within that bucket for multiple hours
  3. Suddenly any attempt to create, or access files just returns an "Input/Output error"
  4. When looking into the logs S3FS just halts after the ache.cpp:AddNoObjectCache(465): add no object cache entry[path=/test12345] line. When it works it creates more log output but in this error state this is the last line written. (Aka there is NO line s3fs.cpp:s3fs_create(1169): [path=/test12345][mode=100644][flags=0x8841] and following written into the logs it fails somewhere in between these two lines)
Originally created by @agowa on GitHub (May 12, 2025). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2670 <!-- -------------------------------------------------------------------------- The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all. --------------------------------------------------------------------------- --> ### Additional Information #### Version of s3fs being used (`s3fs --version`) Amazon Simple Storage Service File System V1.93 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 2.9.9 #### Provider (`AWS`, `OVH`, `Hetzner`, `iDrive E2`, ...) IONOS Cloud #### Kernel information (`uname -r`) 6.14.5-arch1-1 #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) NAME="Arch Linux" PRETTY_NAME="Arch Linux" ID=arch BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://archlinux.org/" DOCUMENTATION_URL="https://wiki.archlinux.org/" SUPPORT_URL="https://bbs.archlinux.org/" BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues" PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/" LOGO=archlinux-logo #### How to run s3fs, if applicable I'm using a systemd mount-unit. However I'm also going to attach the resulting command line just in case. [x] command line: `s3fs 88dbcaef-04bc-4e15-a177-c760a46ca70b /mnt/s3fs -o rw,dbglevel=info,curldbg,logfile=/var/log/s3fs.log,nosuid,nodev,noexec,noatime,allow_other,uid=1000,gid=984,default_permissions,passwd_file=/home/user/.passwd-s3fs,url=https://s3.eu-central-3.ionoscloud.com,tmpdir=/mnt/s3fs-tmp,use_cache=/mnt/s3fs-cache,max_stat_cache_size=2500,parallel_count=48,ensure_diskfree=10240,host=https://s3.eu-central-3.ionoscloud.com,endpoint=eu-central-3,enable_content_md5` [] /etc/fstab [x] systemd ``` # /etc/systemd/system/mnt-s3fs.mount [Unit] Description=S3FS Before=remote-fs.target [Mount] What=88dbcaef-04bc-4e15-a177-c760a46ca70b Where=/mnt/s3fs Type=fuse.s3fs Options=dbglevel=info,curldbg,logfile=/var/log/s3fs.log,_netdev,rw,nosuid,nodev,noexec,noatime,allow_other,uid=1000,gid=984,default_permissions,passwd_file=/home/user/.passwd-s3fs,url=https://s3.eu-central-3.ionoscloud.com,tmpdir=/mnt/s3fs-tmp,use_cache=/mnt/s3fs-cache,max_stat_cache_size=2500,parallel_count=48,ensure_diskfree=10240,host=https://s3.eu-central-3.ionoscloud.com,endpoint=eu-central-3,enable_content_md5 [Install] WantedBy=remote-fs.target WantedBy=multi-user.target ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) <!-- if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages. --> ``` 2025-05-12T06:53:26.234Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/] 2025-05-12T06:53:26.236Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/test12345] 2025-05-12T06:53:26.236Z [INF] curl.cpp:HeadRequest(3299): [tpath=/test12345] 2025-05-12T06:53:26.236Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/test12345][bpath=][save=][sseckeypos=<<<removed>>>] 2025-05-12T06:53:26.236Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345 2025-05-12T06:53:26.236Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345 2025-05-12T06:53:26.236Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345] [] [] 2025-05-12T06:53:26.236Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.236Z [CURL DBG] * Connection 10 seems to be dead 2025-05-12T06:53:26.236Z [CURL DBG] * shutting down connection #10 2025-05-12T06:53:26.236Z [CURL DBG] * Hostname 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com was found in DNS cache 2025-05-12T06:53:26.236Z [CURL DBG] * Trying [2a01:239:100:4ff::fffe]:443... 2025-05-12T06:53:26.255Z [CURL DBG] * SSL reusing session with ALPN '-' 2025-05-12T06:53:26.256Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Client hello (1): 2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Server hello (2): 2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS change cipher, Change cipher spec (1): 2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): 2025-05-12T06:53:26.273Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Certificate (11): 2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, CERT verify (15): 2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Finished (20): 2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): 2025-05-12T06:53:26.275Z [CURL DBG] * TLSv1.3 (OUT), TLS handshake, Finished (20): 2025-05-12T06:53:26.275Z [CURL DBG] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey 2025-05-12T06:53:26.275Z [CURL DBG] * Server certificate: 2025-05-12T06:53:26.275Z [CURL DBG] * subject: C=DE; ST=Rheinland-Pfalz; O=IONOS SE; CN=s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.275Z [CURL DBG] * start date: Feb 27 00:00:00 2025 GMT 2025-05-12T06:53:26.275Z [CURL DBG] * expire date: Feb 27 23:59:59 2026 GMT 2025-05-12T06:53:26.275Z [CURL DBG] * subjectAltName: host "88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com" matched cert's "*.s3.eu-central-3.ionoscloud.com" 2025-05-12T06:53:26.275Z [CURL DBG] * issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Organization Validation Secure Server CA 2025-05-12T06:53:26.275Z [CURL DBG] * SSL certificate verify ok. 2025-05-12T06:53:26.275Z [CURL DBG] * Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256 2025-05-12T06:53:26.275Z [CURL DBG] * Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384 2025-05-12T06:53:26.275Z [CURL DBG] * Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384 2025-05-12T06:53:26.275Z [CURL DBG] * Connected to 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com (2a01:239:100:4ff::fffe) port 443 2025-05-12T06:53:26.275Z [CURL DBG] * using HTTP/1.x 2025-05-12T06:53:26.275Z [CURL DBG] > HEAD /test12345 HTTP/1.1 2025-05-12T06:53:26.275Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.275Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL) 2025-05-12T06:53:26.275Z [CURL DBG] > Accept: */* 2025-05-12T06:53:26.275Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>> 2025-05-12T06:53:26.275Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2025-05-12T06:53:26.275Z [CURL DBG] > x-amz-date: 20250512T065326Z 2025-05-12T06:53:26.275Z [CURL DBG] > 2025-05-12T06:53:26.275Z [CURL DBG] * Request completely sent off 2025-05-12T06:53:26.291Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 2025-05-12T06:53:26.291Z [CURL DBG] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): 2025-05-12T06:53:26.294Z [CURL DBG] < HTTP/1.1 404 Not Found 2025-05-12T06:53:26.294Z [CURL DBG] < content-length: 283 2025-05-12T06:53:26.294Z [CURL DBG] < x-amz-request-id: tx000001495420fd3f39f63-0068219ae6-11d1209-eu-central-3 2025-05-12T06:53:26.294Z [CURL DBG] < accept-ranges: bytes 2025-05-12T06:53:26.294Z [CURL DBG] < content-type: application/xml 2025-05-12T06:53:26.294Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT 2025-05-12T06:53:26.294Z [CURL DBG] < 2025-05-12T06:53:26.294Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact 2025-05-12T06:53:26.294Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT 2025-05-12T06:53:26.294Z [INF] curl.cpp:HeadRequest(3299): [tpath=/test12345/] 2025-05-12T06:53:26.294Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/test12345/][bpath=][save=][sseckeypos=<<<removed>>>] 2025-05-12T06:53:26.294Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345/ 2025-05-12T06:53:26.294Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345/ 2025-05-12T06:53:26.294Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345/] [] [] 2025-05-12T06:53:26.294Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.294Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.294Z [CURL DBG] > HEAD /test12345/ HTTP/1.1 2025-05-12T06:53:26.294Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.294Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL) 2025-05-12T06:53:26.294Z [CURL DBG] > Accept: */* 2025-05-12T06:53:26.294Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>> 2025-05-12T06:53:26.294Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2025-05-12T06:53:26.294Z [CURL DBG] > x-amz-date: 20250512T065326Z 2025-05-12T06:53:26.294Z [CURL DBG] > 2025-05-12T06:53:26.294Z [CURL DBG] * Request completely sent off 2025-05-12T06:53:26.313Z [CURL DBG] < HTTP/1.1 404 Not Found 2025-05-12T06:53:26.313Z [CURL DBG] < content-length: 283 2025-05-12T06:53:26.313Z [CURL DBG] < x-amz-request-id: tx000000e68cf4acc5f9d1c-0068219ae6-11d11f5-eu-central-3 2025-05-12T06:53:26.313Z [CURL DBG] < accept-ranges: bytes 2025-05-12T06:53:26.313Z [CURL DBG] < content-type: application/xml 2025-05-12T06:53:26.313Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT 2025-05-12T06:53:26.313Z [CURL DBG] < 2025-05-12T06:53:26.313Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact 2025-05-12T06:53:26.313Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT 2025-05-12T06:53:26.313Z [INF] s3fs.cpp:list_bucket(3434): [path=/test12345] 2025-05-12T06:53:26.313Z [INF] curl.cpp:ListBucketRequest(3749): [tpath=/test12345] 2025-05-12T06:53:26.313Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b?delimiter=/&max-keys=2&prefix=test12345/ 2025-05-12T06:53:26.313Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com?delimiter=/&max-keys=2&prefix=test12345/ 2025-05-12T06:53:26.313Z [INF] curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test12345/] [] 2025-05-12T06:53:26.313Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.313Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.313Z [CURL DBG] > GET /?delimiter=/&max-keys=2&prefix=test12345/ HTTP/1.1 2025-05-12T06:53:26.313Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.313Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL) 2025-05-12T06:53:26.313Z [CURL DBG] > Accept: */* 2025-05-12T06:53:26.313Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>> 2025-05-12T06:53:26.313Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2025-05-12T06:53:26.313Z [CURL DBG] > x-amz-date: 20250512T065326Z 2025-05-12T06:53:26.313Z [CURL DBG] > 2025-05-12T06:53:26.313Z [CURL DBG] * Request completely sent off 2025-05-12T06:53:26.336Z [CURL DBG] < HTTP/1.1 200 OK 2025-05-12T06:53:26.336Z [CURL DBG] < transfer-encoding: chunked 2025-05-12T06:53:26.336Z [CURL DBG] < x-amz-request-id: tx000004295d3c06aabc064-0068219ae6-11ce797-eu-central-3 2025-05-12T06:53:26.336Z [CURL DBG] < content-type: application/xml 2025-05-12T06:53:26.336Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT 2025-05-12T06:53:26.336Z [CURL DBG] < 2025-05-12T06:53:26.336Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact 2025-05-12T06:53:26.336Z [INF] curl.cpp:RequestPerform(2520): HTTP response code 200 2025-05-12T06:53:26.336Z [INF] cache.cpp:AddNoObjectCache(465): add no object cache entry[path=/test12345] 2025-05-12T06:53:26.337Z [INF] s3fs.cpp:s3fs_create(1169): [path=/test12345][mode=100644][flags=0x8841] 2025-05-12T06:53:26.337Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/test12345] 2025-05-12T06:53:26.337Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/test12345] 2025-05-12T06:53:26.337Z [ERR] fdcache_entity.cpp:Open(549): failed to open file(/mnt/s3fs-cache/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345). errno(24) 2025-05-12T06:53:26.337Z [ERR] fdcache.cpp:Open(597): failed to open and create new pseudo fd for path(/test12345). 2025-05-12T06:53:26.337Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/test12345] 2025-05-12T06:53:26.337Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/test12345] 2025-05-12T06:53:26.337Z [INF] curl.cpp:HeadRequest(3299): [tpath=/test12345] 2025-05-12T06:53:26.337Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/test12345][bpath=][save=][sseckeypos=<<<removed>>>] 2025-05-12T06:53:26.337Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345 2025-05-12T06:53:26.337Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345 2025-05-12T06:53:26.337Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345] [] [] 2025-05-12T06:53:26.337Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.337Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.337Z [CURL DBG] > HEAD /test12345 HTTP/1.1 2025-05-12T06:53:26.337Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.338Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL) 2025-05-12T06:53:26.338Z [CURL DBG] > Accept: */* 2025-05-12T06:53:26.338Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>> 2025-05-12T06:53:26.338Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2025-05-12T06:53:26.338Z [CURL DBG] > x-amz-date: 20250512T065326Z 2025-05-12T06:53:26.338Z [CURL DBG] > 2025-05-12T06:53:26.338Z [CURL DBG] * Request completely sent off 2025-05-12T06:53:26.355Z [CURL DBG] < HTTP/1.1 404 Not Found 2025-05-12T06:53:26.355Z [CURL DBG] < content-length: 283 2025-05-12T06:53:26.355Z [CURL DBG] < x-amz-request-id: tx000004bac52f28e4365d0-0068219ae6-11d11f5-eu-central-3 2025-05-12T06:53:26.355Z [CURL DBG] < accept-ranges: bytes 2025-05-12T06:53:26.355Z [CURL DBG] < content-type: application/xml 2025-05-12T06:53:26.355Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT 2025-05-12T06:53:26.355Z [CURL DBG] < 2025-05-12T06:53:26.355Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact 2025-05-12T06:53:26.355Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT 2025-05-12T06:53:26.355Z [INF] curl.cpp:HeadRequest(3299): [tpath=/test12345/] 2025-05-12T06:53:26.355Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/test12345/][bpath=][save=][sseckeypos=<<<removed>>>] 2025-05-12T06:53:26.355Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345/ 2025-05-12T06:53:26.355Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com/test12345/ 2025-05-12T06:53:26.355Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/test12345/] [] [] 2025-05-12T06:53:26.355Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.355Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.355Z [CURL DBG] > HEAD /test12345/ HTTP/1.1 2025-05-12T06:53:26.355Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.355Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL) 2025-05-12T06:53:26.355Z [CURL DBG] > Accept: */* 2025-05-12T06:53:26.355Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>> 2025-05-12T06:53:26.355Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2025-05-12T06:53:26.355Z [CURL DBG] > x-amz-date: 20250512T065326Z 2025-05-12T06:53:26.355Z [CURL DBG] > 2025-05-12T06:53:26.355Z [CURL DBG] * Request completely sent off 2025-05-12T06:53:26.373Z [CURL DBG] < HTTP/1.1 404 Not Found 2025-05-12T06:53:26.373Z [CURL DBG] < content-length: 283 2025-05-12T06:53:26.373Z [CURL DBG] < x-amz-request-id: tx00000f884bddfc38c53cd-0068219ae6-11ce797-eu-central-3 2025-05-12T06:53:26.373Z [CURL DBG] < accept-ranges: bytes 2025-05-12T06:53:26.373Z [CURL DBG] < content-type: application/xml 2025-05-12T06:53:26.373Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT 2025-05-12T06:53:26.373Z [CURL DBG] < 2025-05-12T06:53:26.373Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact 2025-05-12T06:53:26.373Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT 2025-05-12T06:53:26.373Z [INF] s3fs.cpp:list_bucket(3434): [path=/test12345] 2025-05-12T06:53:26.373Z [INF] curl.cpp:ListBucketRequest(3749): [tpath=/test12345] 2025-05-12T06:53:26.373Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.eu-central-3.ionoscloud.com/88dbcaef-04bc-4e15-a177-c760a46ca70b?delimiter=/&max-keys=2&prefix=test12345/ 2025-05-12T06:53:26.373Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com?delimiter=/&max-keys=2&prefix=test12345/ 2025-05-12T06:53:26.373Z [INF] curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=test12345/] [] 2025-05-12T06:53:26.373Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.373Z [CURL DBG] * Re-using existing https: connection with host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.373Z [CURL DBG] > GET /?delimiter=/&max-keys=2&prefix=test12345/ HTTP/1.1 2025-05-12T06:53:26.373Z [CURL DBG] > Host: 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com 2025-05-12T06:53:26.373Z [CURL DBG] > User-Agent: s3fs/1.93 (commit hash unknown; OpenSSL) 2025-05-12T06:53:26.373Z [CURL DBG] > Accept: */* 2025-05-12T06:53:26.373Z [CURL DBG] > Authorization: AWS4-HMAC-SHA256 Credential=<<<removed>>/eu-central-3/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<<<removed>>> 2025-05-12T06:53:26.373Z [CURL DBG] > x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2025-05-12T06:53:26.373Z [CURL DBG] > x-amz-date: 20250512T065326Z 2025-05-12T06:53:26.373Z [CURL DBG] > 2025-05-12T06:53:26.373Z [CURL DBG] * Request completely sent off 2025-05-12T06:53:26.392Z [CURL DBG] < HTTP/1.1 200 OK 2025-05-12T06:53:26.392Z [CURL DBG] < transfer-encoding: chunked 2025-05-12T06:53:26.392Z [CURL DBG] < x-amz-request-id: tx000002bf0de7bf57a7436-0068219ae6-11d1209-eu-central-3 2025-05-12T06:53:26.392Z [CURL DBG] < content-type: application/xml 2025-05-12T06:53:26.392Z [CURL DBG] < date: Mon, 12 May 2025 06:53:26 GMT 2025-05-12T06:53:26.392Z [CURL DBG] < 2025-05-12T06:53:26.392Z [CURL DBG] * Connection #11 to host 88dbcaef-04bc-4e15-a177-c760a46ca70b.s3.eu-central-3.ionoscloud.com left intact 2025-05-12T06:53:26.392Z [INF] curl.cpp:RequestPerform(2520): HTTP response code 200 2025-05-12T06:53:26.393Z [INF] cache.cpp:AddNoObjectCache(465): add no object cache entry[path=/test12345] ``` ### Details about issue 1. mount s3bucket 2. Do operations on/within that bucket for multiple hours 3. Suddenly any attempt to create, or access files just returns an "Input/Output error" 4. When looking into the logs S3FS just halts after the `ache.cpp:AddNoObjectCache(465): add no object cache entry[path=/test12345]` line. When it works it creates more log output but in this error state this is the last line written. (Aka there is NO line `s3fs.cpp:s3fs_create(1169): [path=/test12345][mode=100644][flags=0x8841]` and following written into the logs it fails somewhere in between these two lines)
kerem commented 2026-03-04 01:52:44 +03:00
Author
Owner
Copy link

@CarstenGrohmann commented on GitHub (Mar 1, 2026):

The error is likely caused by use_cache exhausting all available file descriptors when reading and writing a large number of files simultaneously.

The log entry points to this:

2025-05-12T06:53:26.337Z [ERR] fdcache_entity.cpp:Open(549): failed to open file(/mnt/s3fs-cache/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345). errno(24)

Error 24 is EMFILE — "Too many open files". With use_cache and parallel_count=48 many concurrent file operations can exceed the default limit of 1024.

Workaround: raise LimitNOFILE in the systemd mount unit:

# Add to mnt-s3fs.mount
[Service]
LimitNOFILE=65536

IMHO: This is normal Unix behaviour, not a bug in s3fs

<!-- gh-comment-id:3979699636 --> @CarstenGrohmann commented on GitHub (Mar 1, 2026): The error is likely caused by use_cache exhausting all available file descriptors when reading and writing a large number of files simultaneously. The log entry points to this: ``` 2025-05-12T06:53:26.337Z [ERR] fdcache_entity.cpp:Open(549): failed to open file(/mnt/s3fs-cache/88dbcaef-04bc-4e15-a177-c760a46ca70b/test12345). errno(24) ``` Error 24 is EMFILE — "Too many open files". With `use_cache` and `parallel_count=48` many concurrent file operations can exceed the default limit of 1024. Workaround: raise LimitNOFILE in the systemd mount unit: ``` # Add to mnt-s3fs.mount [Service] LimitNOFILE=65536 ``` IMHO: This is normal Unix behaviour, not a bug in s3fs
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1274
No description provided.