starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1816] rsync IO error on ftruncate #928

New issue
Closed
opened 2026-03-04 01:50:00 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:50:00 +03:00
Owner
Copy link

Originally created by @skepticalwaves on GitHub (Dec 10, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1816

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.90 (commit:v1.90) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

# dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 113
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 2.9.9-3
Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
 /etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
 Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
 export a virtual filesystem to the Linux kernel. It also aims to provide a
 secure method for non privileged users to create and mount their own filesystem
 implementations.
Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Homepage: https://github.com/libfuse/libfuse/wiki

Kernel information (uname -r)

5.4.0-89-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

s3fs command line used, if applicable

NA

/etc/fstab entry, if applicable

<REDACTED>:/matrix /matrix/s3-media-store/ fuse.s3fs x-systemd.automount,x-systemd.requires=network-online.target,allow_other,url=<REDACTED>,uid=997,gid=1000,use_sse=custom:<REDACTED PATH> 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Rerunning with debug turned on, will have an error once it fails again

Details about issue

Running the command:

rsync -avr --progress --ignore-errors --inplace --size-only --whole-file --ignore-existing --delete /matrix/synapse/storage/media-store /matrix/s3-media-store/

Attempting a file existing on the source:

-rw-r--r-- 1 matrix matrix 23M Dec  2 03:46 /matrix/synapse/storage/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf

Fails to copy:

rsync: ftruncate failed on "/matrix/s3-media-store/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf": Input/output error (5)
Originally created by @skepticalwaves on GitHub (Dec 10, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1816 #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.90 (commit:v1.90) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` # dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 113 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Architecture: amd64 Version: 2.9.9-3 Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-3), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Homepage: https://github.com/libfuse/libfuse/wiki ``` #### Kernel information (uname -r) 5.4.0-89-generic #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="Ubuntu" VERSION="20.04.3 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.3 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal #### s3fs command line used, if applicable ``` NA ``` #### /etc/fstab entry, if applicable ``` <REDACTED>:/matrix /matrix/s3-media-store/ fuse.s3fs x-systemd.automount,x-systemd.requires=network-online.target,allow_other,url=<REDACTED>,uid=997,gid=1000,use_sse=custom:<REDACTED PATH> 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) Rerunning with debug turned on, will have an error once it fails again ### Details about issue Running the command: ``` rsync -avr --progress --ignore-errors --inplace --size-only --whole-file --ignore-existing --delete /matrix/synapse/storage/media-store /matrix/s3-media-store/ ``` Attempting a file existing on the source: ``` -rw-r--r-- 1 matrix matrix 23M Dec 2 03:46 /matrix/synapse/storage/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf ``` Fails to copy: ``` rsync: ftruncate failed on "/matrix/s3-media-store/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf": Input/output error (5) ```
kerem closed this issue 2026-03-04 01:50:00 +03:00
kerem commented 2026-03-04 01:50:01 +03:00
Author
Owner
Copy link

@skepticalwaves commented on GitHub (Dec 10, 2021):

Dec 10 17:58:59 matrix s3fs[48387]: curl.cpp:RequestPerform(2357): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?>
                                    <Error><Code>InvalidRequest</Code><Message>The provided encryption parameters did not match the ones used originally.</Message><RequestId>95F296716AB995A6</RequestId><HostId>9uL8MKvQ5X12MnBrT6NsauYIHjA6gVNZ+kyqBiv1dhiBnEcY3gSFC6QaDeBRLjViVoMgfUUAmQAI</HostId></Error>
Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiPerform(172): thread terminated with non-zero return code: -5
Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiPerform(172): thread terminated with non-zero return code: -5
Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiRead(205): failed a request(400: https://<REDACTED>/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=1&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3)
Dec 10 17:58:59 matrix s3fs[48387]:       [tpath=/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf][start=0][size=10485760][part=1]
Dec 10 17:58:59 matrix s3fs[48387]:       URL is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=1&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3
Dec 10 17:58:59 matrix s3fs[48387]:       URL changed is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=1&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3
Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiRead(205): failed a request(400: https://<REDACTED>/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=2&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3)
Dec 10 17:58:59 matrix s3fs[48387]:       [tpath=/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf][start=10485760][size=13318923][part=2]
Dec 10 17:58:59 matrix s3fs[48387]:       URL is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=2&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3
Dec 10 17:58:59 matrix s3fs[48387]:       URL changed is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=2&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3

Looks like a problem with multipart upload?

<!-- gh-comment-id:991201516 --> @skepticalwaves commented on GitHub (Dec 10, 2021): ``` Dec 10 17:58:59 matrix s3fs[48387]: curl.cpp:RequestPerform(2357): HTTP response code 400, returning EIO. Body Text: <?xml version="1.0" encoding="UTF-8"?> <Error><Code>InvalidRequest</Code><Message>The provided encryption parameters did not match the ones used originally.</Message><RequestId>95F296716AB995A6</RequestId><HostId>9uL8MKvQ5X12MnBrT6NsauYIHjA6gVNZ+kyqBiv1dhiBnEcY3gSFC6QaDeBRLjViVoMgfUUAmQAI</HostId></Error> Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiPerform(172): thread terminated with non-zero return code: -5 Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiPerform(172): thread terminated with non-zero return code: -5 Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiRead(205): failed a request(400: https://<REDACTED>/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=1&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3) Dec 10 17:58:59 matrix s3fs[48387]: [tpath=/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf][start=0][size=10485760][part=1] Dec 10 17:58:59 matrix s3fs[48387]: URL is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=1&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3 Dec 10 17:58:59 matrix s3fs[48387]: URL changed is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=1&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3 Dec 10 17:58:59 matrix s3fs[48387]: curl_multi.cpp:MultiRead(205): failed a request(400: https://<REDACTED>/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=2&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3) Dec 10 17:58:59 matrix s3fs[48387]: [tpath=/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf][start=10485760][size=13318923][part=2] Dec 10 17:58:59 matrix s3fs[48387]: URL is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=2&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3 Dec 10 17:58:59 matrix s3fs[48387]: URL changed is https://<REDACTED>/matrix/matrix/media-store/local_content/AG/OA/gjGwNcUKYHaQIqhiIiEf?partNumber=2&uploadId=qjzNa3CiNVy12mIip1oyIw3oW7uXbHszLbeZ2aq9utESTE3gsqTpNwO0JjJ1YDPOKBr3Wdmqtop65LCP7lq8sGTj9JGr9WkDdebTfClMix3NLEs7G7pHPfAb8YO4Ryi3 ``` Looks like a problem with multipart upload?
kerem commented 2026-03-04 01:50:01 +03:00
Author
Owner
Copy link

@skepticalwaves commented on GitHub (Dec 10, 2021):

nomultipart has avoided the failure.

I note this is on the Wasabi S3 implementation.

<!-- gh-comment-id:991296997 --> @skepticalwaves commented on GitHub (Dec 10, 2021): `nomultipart` has avoided the failure. I note this is on the Wasabi S3 implementation.
kerem commented 2026-03-04 01:50:01 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 6, 2023):

@skepticalwaves I'm sorry for my late reply.
I created PR #2088 for this issue.
Please try this code if you can, or wait a moment for it to be merged.

<!-- gh-comment-id:1373463074 --> @ggtakec commented on GitHub (Jan 6, 2023): @skepticalwaves I'm sorry for my late reply. I created PR #2088 for this issue. Please try this code if you can, or wait a moment for it to be merged.
kerem commented 2026-03-04 01:50:01 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jan 7, 2023):

@skepticalwaves
The code has been merged, so the bug will be fixed in the next version.
If you're still having issues, please reopen or post another issue.
Thanks in advance for your assistance.

<!-- gh-comment-id:1374369887 --> @ggtakec commented on GitHub (Jan 7, 2023): @skepticalwaves The code has been merged, so the bug will be fixed in the next version. If you're still having issues, please reopen or post another issue. Thanks in advance for your assistance.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#928
No description provided.