starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #2389] Input/output error with Scaleway S3 bucket when trying to write #1173

New issue
Open
opened 2026-03-04 01:51:55 +03:00 by kerem · 8 comments
kerem commented 2026-03-04 01:51:55 +03:00
Owner
Copy link

Originally created by @Doctor-Who on GitHub (Dec 9, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2389

Additional Information

Version of s3fs being used (s3fs --version)

V1.93 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

2.9.9

Kernel information (uname -r)

6.6.3-1-default

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="openSUSE Tumbleweed"
# VERSION="20231207"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20231207"
PRETTY_NAME="openSUSE Tumbleweed"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20231207"
BUG_REPORT_URL="https://bugzilla.opensuse.org"
SUPPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org"
DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed"
LOGO="distributor-logo-Tumbleweed"

How to run s3fs, if applicable

Used in fstab as below :

s3fs#reolink /srv/backup/reolink fuse _netdev,allow_other,use_path_request_style,passwd_file=/root/.passwd-s3fs,url=https://s3.fr-par.scw.cloud/ 0 0
s3fs#gislab-homeassistant /srv/backup/homeassistant fuse _netdev,allow_other,use_path_request_style,passwd_file=/root/.passwd-s3fs,url=https://s3.fr-par.scw.cloud/ 0 0

cp files or even a simple touch text file :
touch: failed to close 'test.txt': Input/output error
cp: cannot create directory '/srv/backup/reolink/2023/12': Input/output error

information : rm command well works

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Cannot find any log, I'm running on a Pi3B+ :/

Details about issue

Since few weeks, upload to my s3 bucket is not working anymore. Was working during 2 years without any problem.
It is suppose to copy from FTP (camera video) to S3 mount point to send them into the cloud.
I've try to umount and mount without any success.
Even rebooting the Raspberry not working.

I've see other issue but without any solution (or any solution I cannot understand)

Thanks for this amazing opensource project.

Originally created by @Doctor-Who on GitHub (Dec 9, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2389 ### Additional Information #### Version of s3fs being used (`s3fs --version`) V1.93 (commit:unknown) with OpenSSL #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) 2.9.9 #### Kernel information (`uname -r`) 6.6.3-1-default #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) ``` NAME="openSUSE Tumbleweed" # VERSION="20231207" ID="opensuse-tumbleweed" ID_LIKE="opensuse suse" VERSION_ID="20231207" PRETTY_NAME="openSUSE Tumbleweed" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:opensuse:tumbleweed:20231207" BUG_REPORT_URL="https://bugzilla.opensuse.org" SUPPORT_URL="https://bugs.opensuse.org" HOME_URL="https://www.opensuse.org" DOCUMENTATION_URL="https://en.opensuse.org/Portal:Tumbleweed" LOGO="distributor-logo-Tumbleweed" ``` #### How to run s3fs, if applicable Used in fstab as below : `s3fs#reolink /srv/backup/reolink fuse _netdev,allow_other,use_path_request_style,passwd_file=/root/.passwd-s3fs,url=https://s3.fr-par.scw.cloud/ 0 0` `s3fs#gislab-homeassistant /srv/backup/homeassistant fuse _netdev,allow_other,use_path_request_style,passwd_file=/root/.passwd-s3fs,url=https://s3.fr-par.scw.cloud/ 0 0` <!-- Executed command line or /etc/fastab entry --> cp files or even a simple touch text file : `touch: failed to close 'test.txt': Input/output error` `cp: cannot create directory '/srv/backup/reolink/2023/12': Input/output error` information : rm command well works #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`) Cannot find any log, I'm running on a Pi3B+ :/ ### Details about issue Since few weeks, upload to my s3 bucket is not working anymore. Was working during 2 years without any problem. It is suppose to copy from FTP (camera video) to S3 mount point to send them into the cloud. I've try to umount and mount without any success. Even rebooting the Raspberry not working. I've see other issue but without any solution (or any solution I cannot understand) Thanks for this amazing opensource project.
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@klausbadelt commented on GitHub (Dec 9, 2023):

similar issue on AWS S3, simple cp fails after ~50 secs.

<!-- gh-comment-id:1848715254 --> @klausbadelt commented on GitHub (Dec 9, 2023): similar issue on AWS S3, simple `cp` fails after ~50 secs.
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Feb 12, 2024):

@Doctor-Who
The version of s3fs described in this issue is 1.93, but does the fact that it has been working for several years mean that s3fs was upgraded when it stopped working?

I need to know the reason why it doesn't work in detail, so if you can, please specify options such as dbglevel or curldbg and provide the log of the problem area.

Thanks in advance for your assistance.

<!-- gh-comment-id:1938257408 --> @ggtakec commented on GitHub (Feb 12, 2024): @Doctor-Who The version of s3fs described in this issue is 1.93, but does the fact that it has been working for several years mean that s3fs was upgraded when it stopped working? I need to know the reason why it doesn't work in detail, so if you can, please specify options such as `dbglevel` or `curldbg` and provide the log of the problem area. Thanks in advance for your assistance.
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@Doctor-Who commented on GitHub (Feb 25, 2024):

@ggtakec Thanks for your message.
It happens when my openSUSE Tumbleweed updated. It do upgrade every month by cron event. And it stops working since snapshot in december.

Now my version is :

NAME="openSUSE Tumbleweed"
# VERSION="20240131"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20240131"
PRETTY_NAME="openSUSE Tumbleweed"

Perhaps kernel incompatibility ?

<!-- gh-comment-id:1963062156 --> @Doctor-Who commented on GitHub (Feb 25, 2024): @ggtakec Thanks for your message. It happens when my openSUSE Tumbleweed updated. It do upgrade every month by cron event. And it stops working since snapshot in december. Now my version is : ``` NAME="openSUSE Tumbleweed" # VERSION="20240131" ID="opensuse-tumbleweed" ID_LIKE="opensuse suse" VERSION_ID="20240131" PRETTY_NAME="openSUSE Tumbleweed" ``` Perhaps kernel incompatibility ?
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@juliogonzalez commented on GitHub (Feb 25, 2024):

Perhaps kernel incompatibility ?

Please provide the info requested by @ggtakec :-)

It would probably also help if you can list the differences in the installed packages packages, if you still have the right btrtfs snapshots available:

  • One where everything was working
  • One right after it stopped working

One of the runtime requirements for s3fs is fuse. But fuse didn't get a bump in December according to https://build.opensuse.org/package/show/openSUSE%3AFactory/fuse, so could be the kernel or something else.

And s3fs 1.93 was submitted 7 months ago, so the build dependencies should not be a problem either.

<!-- gh-comment-id:1963065491 --> @juliogonzalez commented on GitHub (Feb 25, 2024): > Perhaps kernel incompatibility ? Please provide the info requested by @ggtakec :-) It would probably also help if you can list the differences in the installed packages packages, if you still have the right btrtfs snapshots available: - One where everything was working - One right after it stopped working One of the runtime requirements for s3fs is `fuse`. But `fuse` didn't get a bump in December according to https://build.opensuse.org/package/show/openSUSE%3AFactory/fuse, so could be the kernel or something else. And s3fs 1.93 was submitted 7 months ago, so the build dependencies should not be a problem either.
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@andibraeu commented on GitHub (Feb 25, 2024):

Hi,

I have the same problem, but it's a server running on debian (dev-instance at scaleway):

Version of s3fs being used (s3fs --version)

V1.89 (commit:unknown) with GnuTLS(gcrypt) (delivered with debian)
V1.94 (commit:70a30d6) with OpenSSL

GNU/Linux Distribution, if applicable (cat /etc/os-release)

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel information (uname -r)

4.19.0-26-amd64

What I know: Writing worked until Feb 16, 8:32 UTC, as I have a cronjob running there updating a timestamp every minute. I also don't think there was an update of s3fs. But maybe there were changes at scaleway?

<!-- gh-comment-id:1963091237 --> @andibraeu commented on GitHub (Feb 25, 2024): Hi, I have the same problem, but it's a server running on debian (dev-instance at scaleway): **Version of s3fs being used (s3fs --version)** V1.89 (commit:unknown) with GnuTLS(gcrypt) (delivered with debian) V1.94 (commit:70a30d6) with OpenSSL **GNU/Linux Distribution, if applicable (cat /etc/os-release)** ``` PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` **Kernel information (uname -r)** 4.19.0-26-amd64 What I know: Writing worked until Feb 16, 8:32 UTC, as I have a cronjob running there updating a timestamp every minute. I also don't think there was an update of s3fs. But maybe there were changes at scaleway?
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@andibraeu commented on GitHub (Feb 26, 2024):

I found the root case in the logs:

curl.cpp:RequestPerform(2536): HTTP response code 400, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?>#012<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'</Message><Region>fr-par</Region><RequestId>...</RequestId></Error>

Maybe Scaleway accepted other regions, earlier. And now they updated or corrected their security config. I simply appended endpoint=fr-par to my config and now it works.

<!-- gh-comment-id:1963490193 --> @andibraeu commented on GitHub (Feb 26, 2024): I found the root case in the logs: ``` curl.cpp:RequestPerform(2536): HTTP response code 400, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?>#012<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'fr-par'</Message><Region>fr-par</Region><RequestId>...</RequestId></Error> ``` Maybe Scaleway accepted other regions, earlier. And now they updated or corrected their security config. I simply appended `endpoint=fr-par` to my config and now it works.
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@blacklight commented on GitHub (Feb 27, 2024):

I can confirm that @andibraeu's workaround (adding endpoint=<cc>-<city> to the mount configuration) works.

Same issue here as the others - I can't recall exactly when s3fs stopped working when writing to my Scaleway buckets, but it was probably around November/December.

Maybe worth adding the endpoint= configuration for Scaleway to the README/FAQ?

<!-- gh-comment-id:1967725053 --> @blacklight commented on GitHub (Feb 27, 2024): I can confirm that @andibraeu's workaround (adding `endpoint=<cc>-<city>` to the mount configuration) works. Same issue here as the others - I can't recall exactly when s3fs stopped working when writing to my Scaleway buckets, but it was probably around November/December. Maybe worth adding the `endpoint=` configuration for Scaleway to the README/FAQ?
kerem commented 2026-03-04 01:51:56 +03:00
Author
Owner
Copy link

@andibraeu commented on GitHub (Feb 28, 2024):

I think it's not a workaround, it's the solution ;)

It's also documented by scaleway: https://www.scaleway.com/en/docs/tutorials/object-storage-s3fs/#s3fs-fuse

<!-- gh-comment-id:1968977401 --> @andibraeu commented on GitHub (Feb 28, 2024): I think it's not a workaround, it's the solution ;) It's also documented by scaleway: https://www.scaleway.com/en/docs/tutorials/object-storage-s3fs/#s3fs-fuse
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#1173
No description provided.