[GH-ISSUE #1983] Large file corrupted during upload #1004

New issue

Open

opened 2026-03-04 01:50:34 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:50:34 +03:00

Owner

Originally created by @LesnyRumcajs on GitHub (Jul 14, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1983

Additional Information

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Name        : fuse
Version     : 2.9.9
Release     : 14.fc36
Architecture: x86_64

Kernel information (uname -r)

5.17.5-300.fc36.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="Fedora Linux"
VERSION="36 (Cloud Edition)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora Linux 36 (Cloud Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:36"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=36
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=36
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Cloud Edition"
VARIANT_ID=cloud

s3fs command line used, if applicable

s3fs forest-snapshots /opt/snapshots \
-o passwd_file=${HOME}/.passwd-s3fs \
-o url=https://fra1.digitaloceanspaces.com/ \
-o allow_other \
-d \
-f

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

The debug output for this file was quite massive. I started looking at this after I noticed failures when running it as a daemon (with different multipart_size but it didn't change the behaviour)

Jul 13 09:32:07 forest-test sudo[1278]: chainsafe : TTY=pts/0 ; PWD=/home/chainsafe ; USER=root ; COMMAND=/usr/bin/dnf install s3fs-fuse
Jul 13 09:41:09 forest-test s3fs[1393]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 forest-snapshots /opt/snapshots
Jul 13 09:41:09 forest-test s3fs[1393]: Loaded mime information from /etc/mime.types
Jul 13 09:41:09 forest-test s3fs[1395]: init v1.91(commit:unknown) with OpenSSL
Jul 13 09:41:19 forest-test s3fs[1404]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_path_request_style forest-snapshots /opt/snapshots
Jul 13 09:41:19 forest-test s3fs[1404]: s3fs: MOUNTPOINT directory /opt/snapshots is not empty. if you are sure this is safe, can use the 'nonempty' mount option.
Jul 13 10:15:41 forest-test s3fs[2217]: s3fs version 1.91(unknown) : s3fs umount /opt/snapshots/
Jul 13 10:15:41 forest-test s3fs[2217]: s3fs: MOUNTPOINT directory /opt/snapshots/ is not empty. if you are sure this is safe, can use the 'nonempty' mount option.
Jul 13 10:17:20 forest-test s3fs[2236]: s3fs version 1.91(unknown) : s3fs umount
Jul 13 10:17:20 forest-test s3fs[2236]: Loaded mime information from /etc/mime.types
Jul 13 10:17:20 forest-test s3fs[2236]: s3fs: missing MOUNTPOINT argument.
Jul 13 10:17:56 forest-test s3fs[2237]: s3fs version 1.91(unknown) : s3fs umount lsof
Jul 13 10:17:56 forest-test s3fs[2237]: s3fs: unable to access MOUNTPOINT lsof: No such file or directory
Jul 13 10:25:33 forest-test s3fs[2324]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o allow_other forest-snapshots /opt/snapshots
Jul 13 10:25:33 forest-test s3fs[2324]: Loaded mime information from /etc/mime.types
Jul 13 10:26:10 forest-test s3fs[2334]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o allow_other forest-snapshots /opt/snapshots
Jul 13 10:26:10 forest-test s3fs[2334]: Loaded mime information from /etc/mime.types
Jul 13 10:26:10 forest-test s3fs[2336]: init v1.91(commit:unknown) with OpenSSL                                                                                                                                                                                                            Jul 13 16:23:29 forest-test s3fs[33091]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_path_request_style -o allow_other forest-snapshots /opt/snapshots
Jul 13 16:23:29 forest-test s3fs[33091]: Loaded mime information from /etc/mime.types
Jul 13 16:23:29 forest-test s3fs[33094]: init v1.91(commit:unknown) with OpenSSL
Jul 13 16:34:10 forest-test sudo[33624]: chainsafe : TTY=pts/1 ; PWD=/opt ; USER=root ; COMMAND=/usr/bin/mkdir s3fs_cache
Jul 13 16:34:35 forest-test s3fs[33632]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_cache=/opt/s3fs_cache -o allow_other forest-snapshots /opt/snapshots
Jul 13 16:34:35 forest-test s3fs[33632]: Loaded mime information from /etc/mime.types
Jul 13 16:34:35 forest-test s3fs[33637]: init v1.91(commit:unknown) with OpenSSL
Jul 13 16:36:28 forest-test audit[34540]: USER_CMD pid=34540 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd="/opt/s3fs_cache" cmd=63686F776E202D5220636861696E736166653A636861696E7361666520733366735F6361636865 exe="/usr/bin/sudo" terminal=pts/2 res=success'
Jul 13 16:36:28 forest-test sudo[34540]: chainsafe : TTY=pts/2 ; PWD=/opt/s3fs_cache ; USER=root ; COMMAND=/usr/bin/chown -R chainsafe:chainsafe s3fs_cache
Jul 13 16:36:32 forest-test sudo[34542]: chainsafe : TTY=pts/2 ; PWD=/opt ; USER=root ; COMMAND=/usr/bin/chown -R chainsafe:chainsafe s3fs_cache
Jul 13 16:36:51 forest-test s3fs[34551]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_cache=/opt/s3fs_cache -o allow_other forest-snapshots /opt/snapshots
Jul 13 16:36:51 forest-test s3fs[34551]: Loaded mime information from /etc/mime.types
Jul 13 16:36:51 forest-test s3fs[34553]: init v1.91(commit:unknown) with OpenSSL
Jul 13 16:41:15 forest-test s3fs[35360]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o tmpdir=/opt/s3fs_cache -o allow_other forest-snapshots /opt/snapshots
Jul 13 16:41:15 forest-test s3fs[35360]: Loaded mime information from /etc/mime.types
Jul 13 16:41:15 forest-test s3fs[35363]: init v1.91(commit:unknown) with OpenSSL
Jul 13 16:49:10 forest-test s3fs[36970]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o allow_other forest-snapshots /opt/snapshots
Jul 13 16:49:10 forest-test s3fs[36970]: Loaded mime information from /etc/mime.types
Jul 13 16:49:10 forest-test s3fs[36973]: init v1.91(commit:unknown) with OpenSSL
Jul 13 16:49:30 forest-test sudo[37094]: chainsafe : TTY=pts/2 ; PWD=/opt ; USER=root ; COMMAND=/usr/bin/rmdir s3fs_cache/
Jul 14 07:06:51 forest-test s3fs[137202]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o allow_other forest-snapshots /opt/snapshots
Jul 14 07:06:51 forest-test s3fs[137202]: Loaded mime information from /etc/mime.types
Jul 14 07:06:51 forest-test s3fs[137205]: init v1.91(commit:unknown) with OpenSSL

Details about issue

Uploading a large file (72GB) results in its corruption.

# Verify the checksum of the actual file.
[chainsafe@forest-test real]$ sha256sum minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
2360c2df63f5a7213332acc19ce274ec4c0224e6067b08899c5f544e746b792d  minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
# Print the expected checksum, it matches.
[chainsafe@forest-test real]$ cat minimal_finality_stateroots_1980481_2022-07-13_14-00-30.sha256sum
2360c2df63f5a7213332acc19ce274ec4c0224e6067b08899c5f544e746b792d  -
# Copy the file to the mount.
[chainsafe@forest-test real]$ cp minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car /opt/snapshots/mainnet/
# Check the uploaded file checksum, it's different.
[chainsafe@forest-test real]$ sha256sum /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
1b47eb85f59bbfd5c0d8bbcbcda0ae53f18ad16bc5a36c4adb79e33ed8206021  /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
# Files have the same size
[chainsafe@forest-test real]$ ls -l /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
-rw-r--r--. 1 chainsafe chainsafe 78672352341 Jul 14 09:25 /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
[chainsafe@forest-test real]$ ls -l minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car
-rw-r--r--. 1 chainsafe chainsafe 78672352341 Jul 13 15:16 minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car

I tried with a blank 10G file though and it was fine. It was also fine for a file with a similar structure but of size 4G.

Is there anything I am missing, a flag to provide to the s3fs?

Originally created by @LesnyRumcajs on GitHub (Jul 14, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1983 ### Additional Information #### Version of s3fs being used (s3fs --version) ``` Amazon Simple Storage Service File System V1.91 (commit:unknown) with OpenSSL ``` #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` Name : fuse Version : 2.9.9 Release : 14.fc36 Architecture: x86_64 ``` #### Kernel information (uname -r) ``` 5.17.5-300.fc36.x86_64 ``` #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` NAME="Fedora Linux" VERSION="36 (Cloud Edition)" ID=fedora VERSION_ID=36 VERSION_CODENAME="" PLATFORM_ID="platform:f36" PRETTY_NAME="Fedora Linux 36 (Cloud Edition)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:36" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f36/system-administrators-guide/" SUPPORT_URL="https://ask.fedoraproject.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=36 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=36 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Cloud Edition" VARIANT_ID=cloud ``` #### s3fs command line used, if applicable ``` s3fs forest-snapshots /opt/snapshots \ -o passwd_file=${HOME}/.passwd-s3fs \ -o url=https://fra1.digitaloceanspaces.com/ \ -o allow_other \ -d \ -f ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) The debug output for this file was quite massive. I started looking at this after I noticed failures when running it as a daemon (with different `multipart_size` but it didn't change the behaviour) ``` Jul 13 09:32:07 forest-test sudo[1278]: chainsafe : TTY=pts/0 ; PWD=/home/chainsafe ; USER=root ; COMMAND=/usr/bin/dnf install s3fs-fuse Jul 13 09:41:09 forest-test s3fs[1393]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 forest-snapshots /opt/snapshots Jul 13 09:41:09 forest-test s3fs[1393]: Loaded mime information from /etc/mime.types Jul 13 09:41:09 forest-test s3fs[1395]: init v1.91(commit:unknown) with OpenSSL Jul 13 09:41:19 forest-test s3fs[1404]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_path_request_style forest-snapshots /opt/snapshots Jul 13 09:41:19 forest-test s3fs[1404]: s3fs: MOUNTPOINT directory /opt/snapshots is not empty. if you are sure this is safe, can use the 'nonempty' mount option. Jul 13 10:15:41 forest-test s3fs[2217]: s3fs version 1.91(unknown) : s3fs umount /opt/snapshots/ Jul 13 10:15:41 forest-test s3fs[2217]: s3fs: MOUNTPOINT directory /opt/snapshots/ is not empty. if you are sure this is safe, can use the 'nonempty' mount option. Jul 13 10:17:20 forest-test s3fs[2236]: s3fs version 1.91(unknown) : s3fs umount Jul 13 10:17:20 forest-test s3fs[2236]: Loaded mime information from /etc/mime.types Jul 13 10:17:20 forest-test s3fs[2236]: s3fs: missing MOUNTPOINT argument. Jul 13 10:17:56 forest-test s3fs[2237]: s3fs version 1.91(unknown) : s3fs umount lsof Jul 13 10:17:56 forest-test s3fs[2237]: s3fs: unable to access MOUNTPOINT lsof: No such file or directory Jul 13 10:25:33 forest-test s3fs[2324]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o allow_other forest-snapshots /opt/snapshots Jul 13 10:25:33 forest-test s3fs[2324]: Loaded mime information from /etc/mime.types Jul 13 10:26:10 forest-test s3fs[2334]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o allow_other forest-snapshots /opt/snapshots Jul 13 10:26:10 forest-test s3fs[2334]: Loaded mime information from /etc/mime.types Jul 13 10:26:10 forest-test s3fs[2336]: init v1.91(commit:unknown) with OpenSSL Jul 13 16:23:29 forest-test s3fs[33091]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_path_request_style -o allow_other forest-snapshots /opt/snapshots Jul 13 16:23:29 forest-test s3fs[33091]: Loaded mime information from /etc/mime.types Jul 13 16:23:29 forest-test s3fs[33094]: init v1.91(commit:unknown) with OpenSSL Jul 13 16:34:10 forest-test sudo[33624]: chainsafe : TTY=pts/1 ; PWD=/opt ; USER=root ; COMMAND=/usr/bin/mkdir s3fs_cache Jul 13 16:34:35 forest-test s3fs[33632]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_cache=/opt/s3fs_cache -o allow_other forest-snapshots /opt/snapshots Jul 13 16:34:35 forest-test s3fs[33632]: Loaded mime information from /etc/mime.types Jul 13 16:34:35 forest-test s3fs[33637]: init v1.91(commit:unknown) with OpenSSL Jul 13 16:36:28 forest-test audit[34540]: USER_CMD pid=34540 uid=1000 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='cwd="/opt/s3fs_cache" cmd=63686F776E202D5220636861696E736166653A636861696E7361666520733366735F6361636865 exe="/usr/bin/sudo" terminal=pts/2 res=success' Jul 13 16:36:28 forest-test sudo[34540]: chainsafe : TTY=pts/2 ; PWD=/opt/s3fs_cache ; USER=root ; COMMAND=/usr/bin/chown -R chainsafe:chainsafe s3fs_cache Jul 13 16:36:32 forest-test sudo[34542]: chainsafe : TTY=pts/2 ; PWD=/opt ; USER=root ; COMMAND=/usr/bin/chown -R chainsafe:chainsafe s3fs_cache Jul 13 16:36:51 forest-test s3fs[34551]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o use_cache=/opt/s3fs_cache -o allow_other forest-snapshots /opt/snapshots Jul 13 16:36:51 forest-test s3fs[34551]: Loaded mime information from /etc/mime.types Jul 13 16:36:51 forest-test s3fs[34553]: init v1.91(commit:unknown) with OpenSSL Jul 13 16:41:15 forest-test s3fs[35360]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o tmpdir=/opt/s3fs_cache -o allow_other forest-snapshots /opt/snapshots Jul 13 16:41:15 forest-test s3fs[35360]: Loaded mime information from /etc/mime.types Jul 13 16:41:15 forest-test s3fs[35363]: init v1.91(commit:unknown) with OpenSSL Jul 13 16:49:10 forest-test s3fs[36970]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o multipart_size=1000 -o allow_other forest-snapshots /opt/snapshots Jul 13 16:49:10 forest-test s3fs[36970]: Loaded mime information from /etc/mime.types Jul 13 16:49:10 forest-test s3fs[36973]: init v1.91(commit:unknown) with OpenSSL Jul 13 16:49:30 forest-test sudo[37094]: chainsafe : TTY=pts/2 ; PWD=/opt ; USER=root ; COMMAND=/usr/bin/rmdir s3fs_cache/ Jul 14 07:06:51 forest-test s3fs[137202]: s3fs version 1.91(unknown) : s3fs -o passwd_file=/home/chainsafe/.passwd-s3fs -o url=https://fra1.digitaloceanspaces.com/ -o allow_other forest-snapshots /opt/snapshots Jul 14 07:06:51 forest-test s3fs[137202]: Loaded mime information from /etc/mime.types Jul 14 07:06:51 forest-test s3fs[137205]: init v1.91(commit:unknown) with OpenSSL ``` ### Details about issue Uploading a large file (72GB) results in its corruption. ``` # Verify the checksum of the actual file. [chainsafe@forest-test real]$ sha256sum minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car 2360c2df63f5a7213332acc19ce274ec4c0224e6067b08899c5f544e746b792d minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car # Print the expected checksum, it matches. [chainsafe@forest-test real]$ cat minimal_finality_stateroots_1980481_2022-07-13_14-00-30.sha256sum 2360c2df63f5a7213332acc19ce274ec4c0224e6067b08899c5f544e746b792d - # Copy the file to the mount. [chainsafe@forest-test real]$ cp minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car /opt/snapshots/mainnet/ # Check the uploaded file checksum, it's different. [chainsafe@forest-test real]$ sha256sum /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car 1b47eb85f59bbfd5c0d8bbcbcda0ae53f18ad16bc5a36c4adb79e33ed8206021 /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car # Files have the same size [chainsafe@forest-test real]$ ls -l /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car -rw-r--r--. 1 chainsafe chainsafe 78672352341 Jul 14 09:25 /opt/snapshots/mainnet/minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car [chainsafe@forest-test real]$ ls -l minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car -rw-r--r--. 1 chainsafe chainsafe 78672352341 Jul 13 15:16 minimal_finality_stateroots_1980481_2022-07-13_14-00-30.car ``` I tried with a blank 10G file though and it was fine. It was also fine for a file with a similar structure but of size 4G. Is there anything I am missing, a flag to provide to the `s3fs`?