starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00

[GH-ISSUE #1853] s3fs v1.90 issues not seen with v1.89 #945

New issue

Closed

opened 2026-03-04 01:50:07 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:50:07 +03:00

Owner

Originally created by @UnknownVictim on GitHub (Jan 11, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1853

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

[root@centos-live ~]# s3fs --ver
Amazon Simple Storage Service File System V1.90 (commit:unknown) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[root@centos-live ~]#

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

[root@centos-live ~]# rpm -qi fuse
Name        : fuse
Version     : 2.9.7
Release     : 12.el8
Architecture: x86_64
Install Date: Tue 07 Dec 2021 11:45:21 AM CST
Group       : Unspecified
Size        : 210844
License     : GPL+
Signature   : RSA/SHA256, Mon 01 Jul 2019 04:37:42 PM CDT, Key ID 05b555b38483c65d
Source RPM  : fuse-2.9.7-12.el8.src.rpm
Build Date  : Sat 11 May 2019 09:07:38 AM CDT
Build Host  : x86-02.mbox.centos.org
Relocations : (not relocatable)
Packager    : CentOS Buildsys <bugs@centos.org>
Vendor      : CentOS
URL         : http://fuse.sf.net
Summary     : File System in Userspace (FUSE) v2 utilities
Description :
With FUSE it is possible to implement a fully functional filesystem in a
userspace program. This package contains the FUSE v2 userspace tools to
mount a FUSE filesystem.
[root@centos-live ~]#

Kernel information (uname -r)

[root@centos-live ~]# uname -r
4.18.0-348.2.1.el8_5.x86_64
[root@centos-live ~]#

GNU/Linux Distribution, if applicable (cat /etc/os-release)

[root@centos-live ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
[root@centos-live ~]#

s3fs command line used, if applicable

[root@centos-live ~]# /usr/bin/s3fs ${BUCKET_NAME} ${MOUNTPOINT} -o passwd_file=${S3FS_PASSWD_FILE} -o url=https://${COS_ENDPOINT} -o kernel_cache -o max_stat_cache_size=100000 -o multipart_size=64 -o parallel_count=12 -o multireq_max=30 -o max_background=1000 -o dbglevel=warn -o sigv2
[root@centos-live ~]#

/etc/fstab entry, if applicable

N/A

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Normally, the log only contains a repeat of the s3fs command above.
If I enable debug options, the logs become very large. Will provide if it comes to that.

Setup/Environment Details:

From a CentOS 8 live environment, I am creating Ghost images of physical servers and storing directly to COS via s3fs mount.
Bare Metal server is booted to CentOS 8 live environment.
COS bucket is mounted via s3fs.
Ghost is ran creating an image of the server’s physical hdd(s) and storing the image file(s) directly to the COS mount point created via s3fs.

Details about issue

s3fs v1.89:
No issues. Works fine.

s3fs v1.90:

Image creation times are astronomically slower as compared to v1.89. HDD capture with v1.89 takes ~5 minutes. Same HDD capture with v1.90 takes ~5 hours.
There is a bottleneck being created when using v1.90. With v1.89, I can get over 1Gbps transfer rates, however with v1.90, ~50Mbps is the max.
This bottleneck is only seen when Ghost is creating the image directly to the COS mount point. If I capture the image files to a local disk first and then copy to the COS mount point, the bottleneck is not seen.
The ghost image files created are sometimes corrupted. After working with Broadcom, they report:

These have a common pattern. Each customer’s image file is exactly 4Gb in size. Rest of file is padded with zeros. This is not how ghost creates its image files. So I assume that there was used some file storage preallocation after which file size becomes greater than ghost expects.

If I capture the image files to a local disk, the padding does not happen and the image file is not corrupted. Same is true when using s3fs v1.89.

-rwxr-xr-x. 1 root root 4294943133 Jan  7 20:23 test-capture2.gho  <-- created directly to COS via s3fs-fuse v1.89. Ghost chkimg shows image is good.
-rwxr-xr-x. 1 root root 4294943133 Jan  7 16:16 drive_test-capture.gho <-- created directly to a hdd and copied to COS via s3fs-fuse v1.90. Ghost chkimg shows image is good.
-rwxr-xr-x. 1 root root 4294967296 Jan  7 15:46 test-capture.gho  <-- created directly to COS via s3fs-fuse v1.90. Ghost chkimg fails.
-rwxr-xr-x. 1 root root 4294967296 Dec 21 12:29 32725ce6-3525-4ce7-917a-1992348c335f.gho  <-- created directly to COS via s3fs-fuse v1.90. Ghost chkimg fails.

Note the increased file size when created directly to COS via s3fs-fuse v1.90.

Ideas?

Originally created by @UnknownVictim on GitHub (Jan 11, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1853 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) ``` [root@centos-live ~]# s3fs --ver Amazon Simple Storage Service File System V1.90 (commit:unknown) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. [root@centos-live ~]# ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` [root@centos-live ~]# rpm -qi fuse Name : fuse Version : 2.9.7 Release : 12.el8 Architecture: x86_64 Install Date: Tue 07 Dec 2021 11:45:21 AM CST Group : Unspecified Size : 210844 License : GPL+ Signature : RSA/SHA256, Mon 01 Jul 2019 04:37:42 PM CDT, Key ID 05b555b38483c65d Source RPM : fuse-2.9.7-12.el8.src.rpm Build Date : Sat 11 May 2019 09:07:38 AM CDT Build Host : x86-02.mbox.centos.org Relocations : (not relocatable) Packager : CentOS Buildsys <bugs@centos.org> Vendor : CentOS URL : http://fuse.sf.net Summary : File System in Userspace (FUSE) v2 utilities Description : With FUSE it is possible to implement a fully functional filesystem in a userspace program. This package contains the FUSE v2 userspace tools to mount a FUSE filesystem. [root@centos-live ~]# ``` #### Kernel information (uname -r) ``` [root@centos-live ~]# uname -r 4.18.0-348.2.1.el8_5.x86_64 [root@centos-live ~]# ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` [root@centos-live ~]# cat /etc/os-release NAME="CentOS Linux" VERSION="8" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" PRETTY_NAME="CentOS Linux 8" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:8" HOME_URL="https://centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-8" CENTOS_MANTISBT_PROJECT_VERSION="8" [root@centos-live ~]# ``` #### s3fs command line used, if applicable ``` [root@centos-live ~]# /usr/bin/s3fs ${BUCKET_NAME} ${MOUNTPOINT} -o passwd_file=${S3FS_PASSWD_FILE} -o url=https://${COS_ENDPOINT} -o kernel_cache -o max_stat_cache_size=100000 -o multipart_size=64 -o parallel_count=12 -o multireq_max=30 -o max_background=1000 -o dbglevel=warn -o sigv2 [root@centos-live ~]# ``` #### /etc/fstab entry, if applicable N/A #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) Normally, the log only contains a repeat of the s3fs command above. If I enable debug options, the logs become very large. Will provide if it comes to that. #### Setup/Environment Details: From a CentOS 8 live environment, I am creating Ghost images of physical servers and storing directly to COS via s3fs mount. Bare Metal server is booted to CentOS 8 live environment. COS bucket is mounted via s3fs. Ghost is ran creating an image of the server’s physical hdd(s) and storing the image file(s) directly to the COS mount point created via s3fs. #### Details about issue **s3fs v1.89:** No issues. Works fine. **s3fs v1.90:** 1. Image creation times are astronomically slower as compared to v1.89. HDD capture with v1.89 takes ~5 minutes. Same HDD capture with v1.90 takes ~5 hours. There is a bottleneck being created when using v1.90. With v1.89, I can get over 1Gbps transfer rates, however with v1.90, ~50Mbps is the max. This bottleneck is only seen when Ghost is creating the image directly to the COS mount point. If I capture the image files to a local disk first and then copy to the COS mount point, the bottleneck is not seen. 2. The ghost image files created are sometimes corrupted. After working with Broadcom, they report: > These have a common pattern. Each customer’s image file is exactly 4Gb in size. Rest of file is padded with zeros. This is not how ghost creates its image files. So I assume that there was used some file storage preallocation after which file size becomes greater than ghost expects. If I capture the image files to a local disk, the padding does not happen and the image file is not corrupted. Same is true when using s3fs v1.89. ``` -rwxr-xr-x. 1 root root 4294943133 Jan 7 20:23 test-capture2.gho <-- created directly to COS via s3fs-fuse v1.89. Ghost chkimg shows image is good. -rwxr-xr-x. 1 root root 4294943133 Jan 7 16:16 drive_test-capture.gho <-- created directly to a hdd and copied to COS via s3fs-fuse v1.90. Ghost chkimg shows image is good. -rwxr-xr-x. 1 root root 4294967296 Jan 7 15:46 test-capture.gho <-- created directly to COS via s3fs-fuse v1.90. Ghost chkimg fails. -rwxr-xr-x. 1 root root 4294967296 Dec 21 12:29 32725ce6-3525-4ce7-917a-1992348c335f.gho <-- created directly to COS via s3fs-fuse v1.90. Ghost chkimg fails. ``` Note the increased file size when created directly to COS via s3fs-fuse v1.90. Ideas?

kerem

2026-03-04 01:50:07 +03:00

closed this issue
added the
need info

dataloss
labels

kerem commented

2026-03-04 01:50:08 +03:00

Author

Owner

@gaul commented on GitHub (Jan 24, 2022):

Do you know how to git bisect? If so, you can narrow down to which commit introduced this regression.

Another approach would be to try to disable features to see if one works around the issue, e.g., -o max_dirty_data=-1, -o nomixupload, -o nomultipart. This would also help narrow down the issue.

Finally it would help to dump the s3fs logs via s3fs -f -d -o curldbg so we can try to write a test that demonstrates the issue.

@gaul commented on GitHub (Jan 24, 2022): Do you know how to `git bisect`? If so, you can narrow down to which commit introduced this regression. Another approach would be to try to disable features to see if one works around the issue, e.g., `-o max_dirty_data=-1`, `-o nomixupload`, `-o nomultipart`. This would also help narrow down the issue. Finally it would help to dump the s3fs logs via `s3fs -f -d -o curldbg` so we can try to write a test that demonstrates the issue.

kerem commented

2026-03-04 01:50:08 +03:00

Author

Owner

@gaul commented on GitHub (Sep 8, 2023):

Please test with 1.93 and reopen if your symptoms persist.

@gaul commented on GitHub (Sep 8, 2023): Please test with 1.93 and reopen if your symptoms persist.

kerem referenced this issue

2026-03-04 02:01:51 +03:00

[PR #951] [MERGED] Added a non-interactive option to utility mode #1723