starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1592] Unable to mv or cp big files to object storage, which is mounted with s3-fs #835

New issue
Closed
opened 2026-03-04 01:49:12 +03:00 by kerem · 4 comments
kerem commented 2026-03-04 01:49:12 +03:00
Owner
Copy link

Originally created by @erki-karblane on GitHub (Mar 3, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1592

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.89 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Version : 2.9.2

Kernel information (uname -r)

3.10.0-1160.15.2.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="7 (Core)"

s3fs command line used, if applicable

/etc/fstab entry, if applicable

s3fs#application-backup /mnt/application-backup fuse _netdev,allow_other,url=https://s3.private.eu-de.cloud-object-storage.appdomain.cloud 0 0

#### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)
_if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_
Mar 03 08:14:59 amsssobbva.asc-hs.cloud yum[6456]: Updated: s3fs-fuse-1.89-1.el7.x86_64
Mar 03 08:17:06 amsssobbva.asc-hs.cloud s3fs[6756]: Loaded mime information from /etc/mime.types
Mar 03 08:17:06 amsssobbva.asc-hs.cloud s3fs[6757]: init v1.89(commit:unknown) with OpenSSL

Details about issue

I am trying to move large files (over 80G) to object storeage, but I get I/O errors and the file size in the object storage is always 5368709120.
Tried with version 1.88 and 1.89

mv errors:
mv: preserving times for ‘/mnt/application-backup/xxx/test7.zip’: Operation not permitted
mv: preserving permissions for ‘/mnt/application-backup/xxx/test7.zip’: Operation not permitted
mv: failed to close ‘/mnt/application-backup/xxx/test7.zip’: Input/output error

cp errors:
cp: failed to close ‘/mnt/application-backup/xxx/test6.zip’: Input/output error

Originally created by @erki-karblane on GitHub (Mar 3, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1592 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.89 (commit:unknown) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Version : 2.9.2 #### Kernel information (uname -r) 3.10.0-1160.15.2.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="CentOS Linux" VERSION="7 (Core)" #### s3fs command line used, if applicable ``` ``` #### /etc/fstab entry, if applicable s3fs#application-backup /mnt/application-backup fuse _netdev,allow_other,url=https://s3.private.eu-de.cloud-object-storage.appdomain.cloud 0 0 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ Mar 03 08:14:59 amsssobbva.asc-hs.cloud yum[6456]: Updated: s3fs-fuse-1.89-1.el7.x86_64 Mar 03 08:17:06 amsssobbva.asc-hs.cloud s3fs[6756]: Loaded mime information from /etc/mime.types Mar 03 08:17:06 amsssobbva.asc-hs.cloud s3fs[6757]: init v1.89(commit:unknown) with OpenSSL ``` ### Details about issue I am trying to move large files (over 80G) to object storeage, but I get I/O errors and the file size in the object storage is always 5368709120. Tried with version 1.88 and 1.89 mv errors: mv: preserving times for ‘/mnt/application-backup/xxx/test7.zip’: Operation not permitted mv: preserving permissions for ‘/mnt/application-backup/xxx/test7.zip’: Operation not permitted mv: failed to close ‘/mnt/application-backup/xxx/test7.zip’: Input/output error cp errors: cp: failed to close ‘/mnt/application-backup/xxx/test6.zip’: Input/output error
kerem 2026-03-04 01:49:12 +03:00
  • closed this issue
  • added the
    need info
         label
kerem commented 2026-03-04 01:49:13 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Mar 3, 2021):

Can you run s3fs with -f -o curldbg which may reveal the cause? Do you use AWS or some other object store? If the latter, perhaps it does not support MPU CopyPart? As a temporary workaround you might try setting -o max_dirty_data=-1.

<!-- gh-comment-id:789667360 --> @gaul commented on GitHub (Mar 3, 2021): Can you run s3fs with `-f -o curldbg` which may reveal the cause? Do you use AWS or some other object store? If the latter, perhaps it does not support MPU CopyPart? As a temporary workaround you might try setting `-o max_dirty_data=-1`.
kerem commented 2026-03-04 01:49:14 +03:00
Author
Owner
Copy link

@erki-karblane commented on GitHub (Mar 3, 2021):

Thanks, the -o max_dirty_data=-1 seems to be working.

<!-- gh-comment-id:789761327 --> @erki-karblane commented on GitHub (Mar 3, 2021): Thanks, the -o max_dirty_data=-1 seems to be working.
kerem commented 2026-03-04 01:49:14 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Mar 4, 2021):

While this workaround may work, you may want to root cause it. Again, do you use AWS or some other object store? -o nocopyapi may be the more appropriate flag if you have a similar problem when renaming.

<!-- gh-comment-id:790199186 --> @gaul commented on GitHub (Mar 4, 2021): While this workaround may work, you may want to root cause it. Again, do you use AWS or some other object store? `-o nocopyapi` may be the more appropriate flag if you have a similar problem when renaming.
kerem commented 2026-03-04 01:49:14 +03:00
Author
Owner
Copy link

@erki-karblane commented on GitHub (Mar 4, 2021):

IBM cloud Object Storage

<!-- gh-comment-id:790420230 --> @erki-karblane commented on GitHub (Mar 4, 2021): IBM cloud Object Storage
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#835
No description provided.