[GH-ISSUE #1328] I/O Error when trying to move a large file

kerem commented

2026-03-04 01:48:06 +03:00

Owner

Originally created by @avaiss on GitHub (Jul 8, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1328

Whenever I am trying to either:

move a large (multipart) file to another directory,
rename a directory containing a large (multipart) file,

the operation fails for I/O error.

Note that does not happen for all files larger than 1G (my configured multipart size), only big ones (19G in my tests).
Also, I am not using Amazon S3, but S3 object storage provided by scaleway

Additional Information

Version of s3fs being used (s3fs --version)

$ s3fs --version
Amazon Simple Storage Service File System V1.84(commit:unknown) with GnuTLS(gcrypt)
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

$ dpkg -s s3fs
Package: s3fs
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 583
Maintainer: Andrii Senkovych <andrii@senkovych.com>
Architecture: amd64
Source: s3fs-fuse
Version: 1.84-1
Depends: fuse, mime-support, libc6 (>= 2.17), libcurl3-gnutls (>= 7.16.2), libfuse2 (>= 2.8), libgcc1 (>= 1:3.0), libgcrypt20 (>= 1.8.0), libgnutls30 (>= 3.5.0), libstdc++6 (>= 5.2), libxml2 (>= 2.7.4)
Description: FUSE-based file system backed by Amazon S3
 s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket
 as a local filesystem. It stores files natively and transparently in
 S3 (i.e., you can use other programs to access the same files).
 Maximum file size=64GB (limited by s3fs, not Amazon).
 .
 s3fs is stable and is being used in a number of production environments,
 e.g., rsync backup to s3.
Homepage: https://github.com/s3fs-fuse/s3fs-fuse

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

$ dpkg -s fuse
Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 140
Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>
Architecture: amd64
Version: 2.9.9-1+deb10u1
Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-1+deb10u1), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
 /etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
 Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
 export a virtual filesystem to the Linux kernel. It also aims to provide a
 secure method for non privileged users to create and mount their own filesystem
 implementations.
Homepage: https://github.com/libfuse/libfuse/wiki

Kernel information (uname -r)

$ uname -r
4.19.0-8-amd64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

$ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

s3fs command line used, if applicable

N/A

/etc/fstab entry, if applicable

xxxxxxxx                            /mnt/s3         fuse.s3fs _netdev,noatime,noauto,allow_other,use_path_request_style,multipart_size=1024,url=https://s3.fr-par.scw.cloud/,endpoint=fr-par,use_cache=/path/to/xxxxxxxx-cache,del_cache,x-systemd.automount,x-systemd.idle-timeout=1m,dbglevel=info,curldbg 0 0

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

s3fs.txt.gz

Details about issue

Reproduce steps.

$ ls -lh run-1
total 79G
-rw-r--r-- 1 root root 1.6G Jun 28 04:46 disk.full.20200628.tar.gpg
...
-rw-r--r-- 1 root root  19G Jun 07 05:33 valyria.full.20200607.tar.gpg


$ ls -lh run-x
total 0

$ mv run-1/valyria.full.20200607.tar.gpg run-x/
mv: cannot move 'run-1/valyria.full.20200607.tar.gpg' to 'run-x/valyria.full.20200607.tar.gpg': Input/output error

$ mv run-1/disk.full.20200628.tar.gpg run-x/

$ mv run-x/disk.full.20200628.tar.gpg run-1/

$ rmdir run-x

$ mv run-1 run-x
mv: cannot move 'run-1' to 'run-x/run-1': Input/output error

# Note the weird message above (we should not copy run-1 to run-x but contents of run-1 into run-x)

$ ls -lh run-x
total 2.5G
-rw-r--r-- 1 root root 1.6G Jun 28 04:46 disk.full.20200628.tar.gpg
...
drwxr-xr-x 1 root root    0 Jun 21 03:53 run-1

# All files till the 19G one were copied, and a weird run-1 directory was created.

Originally created by @avaiss on GitHub (Jul 8, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1328 Whenever I am trying to either: * move a large (multipart) file to another directory, * rename a directory containing a large (multipart) file, the operation fails for I/O error. Note that does not happen for all files larger than 1G (my configured multipart size), only big ones (19G in my tests). Also, I am not using Amazon S3, but S3 object storage provided by [scaleway](https://www.scaleway.com/) ### Additional Information #### Version of s3fs being used (s3fs --version) ``` $ s3fs --version Amazon Simple Storage Service File System V1.84(commit:unknown) with GnuTLS(gcrypt) Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` ``` $ dpkg -s s3fs Package: s3fs Status: install ok installed Priority: optional Section: utils Installed-Size: 583 Maintainer: Andrii Senkovych <andrii@senkovych.com> Architecture: amd64 Source: s3fs-fuse Version: 1.84-1 Depends: fuse, mime-support, libc6 (>= 2.17), libcurl3-gnutls (>= 7.16.2), libfuse2 (>= 2.8), libgcc1 (>= 1:3.0), libgcrypt20 (>= 1.8.0), libgnutls30 (>= 3.5.0), libstdc++6 (>= 5.2), libxml2 (>= 2.7.4) Description: FUSE-based file system backed by Amazon S3 s3fs is a FUSE filesystem that allows you to mount an Amazon S3 bucket as a local filesystem. It stores files natively and transparently in S3 (i.e., you can use other programs to access the same files). Maximum file size=64GB (limited by s3fs, not Amazon). . s3fs is stable and is being used in a number of production environments, e.g., rsync backup to s3. Homepage: https://github.com/s3fs-fuse/s3fs-fuse ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` $ dpkg -s fuse Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 140 Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> Architecture: amd64 Version: 2.9.9-1+deb10u1 Depends: libc6 (>= 2.28), libfuse2 (= 2.9.9-1+deb10u1), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Homepage: https://github.com/libfuse/libfuse/wiki ``` #### Kernel information (uname -r) ``` $ uname -r 4.19.0-8-amd64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` $ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 10 (buster)" NAME="Debian GNU/Linux" VERSION_ID="10" VERSION="10 (buster)" VERSION_CODENAME=buster ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ``` #### s3fs command line used, if applicable N/A #### /etc/fstab entry, if applicable ``` xxxxxxxx /mnt/s3 fuse.s3fs _netdev,noatime,noauto,allow_other,use_path_request_style,multipart_size=1024,url=https://s3.fr-par.scw.cloud/,endpoint=fr-par,use_cache=/path/to/xxxxxxxx-cache,del_cache,x-systemd.automount,x-systemd.idle-timeout=1m,dbglevel=info,curldbg 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) [s3fs.txt.gz](https://github.com/s3fs-fuse/s3fs-fuse/files/4890043/s3fs.txt.gz) ### Details about issue Reproduce steps. ``` $ ls -lh run-1 total 79G -rw-r--r-- 1 root root 1.6G Jun 28 04:46 disk.full.20200628.tar.gpg ... -rw-r--r-- 1 root root 19G Jun 07 05:33 valyria.full.20200607.tar.gpg $ ls -lh run-x total 0 $ mv run-1/valyria.full.20200607.tar.gpg run-x/ mv: cannot move 'run-1/valyria.full.20200607.tar.gpg' to 'run-x/valyria.full.20200607.tar.gpg': Input/output error $ mv run-1/disk.full.20200628.tar.gpg run-x/ $ mv run-x/disk.full.20200628.tar.gpg run-1/ $ rmdir run-x $ mv run-1 run-x mv: cannot move 'run-1' to 'run-x/run-1': Input/output error # Note the weird message above (we should not copy run-1 to run-x but contents of run-1 into run-x) $ ls -lh run-x total 2.5G -rw-r--r-- 1 root root 1.6G Jun 28 04:46 disk.full.20200628.tar.gpg ... drwxr-xr-x 1 root root 0 Jun 21 03:53 run-1 # All files till the 19G one were copied, and a weird run-1 directory was created. ```

kerem

2026-03-04 01:48:06 +03:00

closed this issue
added the
need info
label

kerem commented

2026-03-04 01:48:07 +03:00

Author

Owner

@gaul commented on GitHub (Jul 8, 2020):

Could you try testing with the latest version 1.86 or master? If symptoms persist, please run s3fs with debug logging via -f -d -o curldbg so we can investigate further.

@gaul commented on GitHub (Jul 8, 2020): Could you try testing with the latest version 1.86 or master? If symptoms persist, please run s3fs with debug logging via `-f -d -o curldbg` so we can investigate further.

kerem commented

2026-03-04 01:48:07 +03:00

Author

Owner

@avaiss commented on GitHub (Jul 8, 2020):

Same issue with master (1f796d4).
s3fs.master.txt.gz

@avaiss commented on GitHub (Jul 8, 2020): Same issue with master (1f796d4). [s3fs.master.txt.gz](https://github.com/s3fs-fuse/s3fs-fuse/files/4890633/s3fs.master.txt.gz)

kerem commented

2026-03-04 01:48:07 +03:00

Author

Owner

@gaul commented on GitHub (Jul 8, 2020):

Relevant error from the logs:

[ERR] curl.cpp:RequestPerform(2569): HTTP response code 409, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>OperationAborted</Code><Message>A conflicting conditional operation is currently in progress against this resource. Please try again.</Message><RequestId>tx6646327434bb49b7896f8-005f05ba9c</RequestId></Error>
[WAN] curl.cpp:MultiPerform(4418): thread failed - rc(-5)

It seems like Scaleway has some incompatibility with the S3 protocol. You can work around this with -o nocopyapi until we can debug this further.

@gaul commented on GitHub (Jul 8, 2020): Relevant error from the logs: ``` [ERR] curl.cpp:RequestPerform(2569): HTTP response code 409, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?> <Error><Code>OperationAborted</Code><Message>A conflicting conditional operation is currently in progress against this resource. Please try again.</Message><RequestId>tx6646327434bb49b7896f8-005f05ba9c</RequestId></Error> [WAN] curl.cpp:MultiPerform(4418): thread failed - rc(-5) ``` It seems like Scaleway has some incompatibility with the S3 protocol. You can work around this with `-o nocopyapi` until we can debug this further.

kerem commented

2026-03-04 01:48:07 +03:00

Author

Owner

@avaiss commented on GitHub (Jul 8, 2020):

Thanks. It would seem that -o nocopyapi helps for renaming directories containing large files. Well... at least files are moved somehow but the destination directory does not look as the original one ;)

Doing mv run-1 run-x with run-1

run-1
 |- A
 |- B
 |- C (large)
 |- D (normal)

leads to:

run-x
 |- A
 |- B
 |- run-1
   |- C (large)
   |- D (normal)

It does not work for the first problem (moving individual very large files). As the error message you found made me think of parallelism issues, I mounted the dir with -o parallel_count=1 and moving individual large files works with that option (but is really slow). Moving directories does not.

So I guess -o nocopyapi -o parallel_count=1 is the way to go (but nocopyapi means 40G traffic for moving a 20G file, which is not ideal :/)

@avaiss commented on GitHub (Jul 8, 2020): Thanks. It would seem that ``-o nocopyapi`` helps for renaming directories containing large files. Well... at least files are moved somehow but the destination directory does not look as the original one ;) Doing ``mv run-1 run-x`` with run-1 ``` run-1 |- A |- B |- C (large) |- D (normal) ``` leads to: ``` run-x |- A |- B |- run-1 |- C (large) |- D (normal) ``` It does not work for the first problem (moving individual very large files). As the error message you found made me think of parallelism issues, I mounted the dir with ``-o parallel_count=1`` and moving individual large files works with that option (but is really slow). Moving directories does not. So I guess ``-o nocopyapi -o parallel_count=1`` is the way to go (but nocopyapi means 40G traffic for moving a 20G file, which is not ideal :/)

kerem commented

2026-03-04 01:48:08 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 13, 2020):

@avaiss If you specify nocopyapi, whole object(file) will be downloaded once to localhost and then uploaded.
In other words, the amount of communication will increase as you mentioned.
I think this is difficult to avoid and cannot be mitigated without enabling caching.

It is a matter that you have to specify parallel_count=1, but it may not recur due to other corrections(ex. #1313, #1319).
Please test again with the latest master code, if you allow it.
Thanks in advance for your kindness.

@ggtakec commented on GitHub (Jul 13, 2020): @avaiss If you specify `nocopyapi`, whole object(file) will be downloaded once to localhost and then uploaded. In other words, the amount of communication will increase as you mentioned. I think this is difficult to avoid and cannot be mitigated without enabling caching. It is a matter that you have to specify `parallel_count=1`, but it may not recur due to other corrections(ex. #1313, #1319). Please test again with the latest master code, if you allow it. Thanks in advance for your kindness.

kerem commented

2026-03-04 01:48:08 +03:00

Author

Owner

@avaiss commented on GitHub (Oct 3, 2020):

I did finally have a bit of time to retest, and well it would seem that indeed, with current master( 4623472 ) moving individual large files on scaleway do not require parallel_count=1.

Still have the issue of failure when moving a directory containing a large file though. Here is an extract from
debug output I found interesting.

I was doing mv ice fire, ice being a directory containing one large ice.full.... file.

[ERR] curl.cpp:RequestPerform(2074): HTTP response code 400, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?>
<Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed object size.</Message><RequestId>tx63fdb8a0e0044cfa8b34b-005f78bf70</RequestId></Error>

@avaiss commented on GitHub (Oct 3, 2020): I did finally have a bit of time to retest, and well it would seem that indeed, with current master( 4623472 ) moving individual large files on scaleway do not require `parallel_count=1`. Still have the issue of failure when moving a directory containing a large file though. Here is an extract from [debug output](https://github.com/s3fs-fuse/s3fs-fuse/files/5322287/s3fs.txt) I found interesting. I was doing `mv ice fire`, `ice` being a directory containing one large `ice.full....` file. ``` [ERR] curl.cpp:RequestPerform(2074): HTTP response code 400, returning EIO. Body Text: <?xml version='1.0' encoding='UTF-8'?> <Error><Code>EntityTooLarge</Code><Message>Your proposed upload exceeds the maximum allowed object size.</Message><RequestId>tx63fdb8a0e0044cfa8b34b-005f78bf70</RequestId></Error> ```

kerem commented

2026-03-04 01:48:08 +03:00

Author

Owner

@gaul commented on GitHub (Feb 8, 2021):

@avaiss #1553 may address these symptoms. Can you retest with master?

@gaul commented on GitHub (Feb 8, 2021): @avaiss #1553 may address these symptoms. Can you retest with master?

kerem commented

2026-03-04 01:48:08 +03:00

Author

Owner

@gaul commented on GitHub (Feb 11, 2021):

Potential duplicate of #1547.

@gaul commented on GitHub (Feb 11, 2021): Potential duplicate of #1547.

kerem commented

2026-03-04 01:48:08 +03:00

Author

Owner

@gaul commented on GitHub (Feb 20, 2021):

Please reopen if you can reproduce these symptoms with the latest master.

@gaul commented on GitHub (Feb 20, 2021): Please reopen if you can reproduce these symptoms with the latest master.

Rows
Columns

[GH-ISSUE #1328] I/O Error when trying to move a large file #711

Additional Information

Version of s3fs being used (s3fs --version)

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Kernel information (uname -r)

GNU/Linux Distribution, if applicable (cat /etc/os-release)

s3fs command line used, if applicable

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Details about issue