[GH-ISSUE #2243] File too large when generating files or copy files to aws bucket #1132

New issue

Open

opened 2026-03-04 01:51:38 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:51:38 +03:00

Owner

Originally created by @chunxuan-hs on GitHub (Jul 30, 2023).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2243

Additional Information

Version of s3fs being used (`s3fs --version`)

Amazon Simple Storage Service File System V1.92 (commit:4b3e715) with OpenSSL
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`)

Package: fuse
Status: install ok installed
Priority: optional
Section: utils
Installed-Size: 114
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 2.9.9-5ubuntu3
Depends: libc6 (>= 2.34), libfuse2 (= 2.9.9-5ubuntu3), adduser, mount (>= 2.19.1), sed (>= 4)
Conffiles:
 /etc/fuse.conf 298587592c8444196833f317def414f2
Description: Filesystem in Userspace
 Filesystem in Userspace (FUSE) is a simple interface for userspace programs to
 export a virtual filesystem to the Linux kernel. It also aims to provide a
 secure method for non privileged users to create and mount their own filesystem
 implementations.
Homepage: https://github.com/libfuse/libfuse/wiki
Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org>

Kernel information (`uname -r`)

5.19.0-1029-aws

GNU/Linux Distribution, if applicable (`cat /etc/os-release`)

PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

How to run s3fs, if applicable

[x] command line
[] /etc/fstab

s3fs -o compat_dir -o nomultipart bucket_name local_dir

s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)

Jul 30 13:44:28 ip-10-82-70-36 s3fs[604593]: s3fs version 1.92(4b3e715) : s3fs -o compat_dir -o nocopyapi hotspottx-idalab s3b_idalab
Jul 30 13:44:28 ip-10-82-70-36 s3fs[604593]: Loaded mime information from /etc/mime.types
Jul 30 13:44:28 ip-10-82-70-36 s3fs[604601]: s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function.
Jul 30 13:44:28 ip-10-82-70-36 s3fs[604601]: init v1.92(commit:4b3e715) with OpenSSL, credential-library(built-in)
Jul 30 15:19:08 ip-10-82-70-36 kernel: [109986.619304] s3fs[604601]: segfault at 7fda49dff910 ip 00007fda4c49659b sp 00007ffeb34dec30 error 4 in libc.so.6[7fda4c428000+195000]
Jul 30 15:19:43 ip-10-82-70-36 s3fs[628180]: s3fs version 1.92(4b3e715) : s3fs -o compat_dir -o nomultipart hotspottx-idalab s3b_idalab
Jul 30 15:19:43 ip-10-82-70-36 s3fs[628180]: Loaded mime information from /etc/mime.types
Jul 30 15:19:43 ip-10-82-70-36 s3fs[628188]: s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function.
Jul 30 15:19:43 ip-10-82-70-36 s3fs[628188]: init v1.92(commit:4b3e715) with OpenSSL, credential-library(built-in)
Jul 30 16:22:04 ip-10-82-70-36 s3fs[629784]: s3fs version 1.92(4b3e715) : s3fs outputs
Jul 30 16:22:04 ip-10-82-70-36 s3fs[629784]: Loaded mime information from /etc/mime.types
Jul 30 16:22:04 ip-10-82-70-36 s3fs[629784]: s3fs: missing MOUNTPOINT argument.

Details about issue

Tried to cat file1 file2 > file3, and got the error of "File too large". Tried to generate file3 first and copy to bucket via rsync, got the same error.

Initially I used s3fs -o compat_dir bucket_name local_dir, and played a few other options but didn't work.

Any suggestions? Thanks!

Originally created by @chunxuan-hs on GitHub (Jul 30, 2023). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2243  ### Additional Information #### Version of s3fs being used (`s3fs --version`) ``` Amazon Simple Storage Service File System V1.92 (commit:4b3e715) with OpenSSL Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` #### Version of fuse being used (`pkg-config --modversion fuse`, `rpm -qi fuse` or `dpkg -s fuse`) ``` Package: fuse Status: install ok installed Priority: optional Section: utils Installed-Size: 114 Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> Architecture: amd64 Version: 2.9.9-5ubuntu3 Depends: libc6 (>= 2.34), libfuse2 (= 2.9.9-5ubuntu3), adduser, mount (>= 2.19.1), sed (>= 4) Conffiles: /etc/fuse.conf 298587592c8444196833f317def414f2 Description: Filesystem in Userspace Filesystem in Userspace (FUSE) is a simple interface for userspace programs to export a virtual filesystem to the Linux kernel. It also aims to provide a secure method for non privileged users to create and mount their own filesystem implementations. Homepage: https://github.com/libfuse/libfuse/wiki Original-Maintainer: Laszlo Boszormenyi (GCS) <gcs@debian.org> ``` #### Kernel information (`uname -r`) `5.19.0-1029-aws` #### GNU/Linux Distribution, if applicable (`cat /etc/os-release`) ``` PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` #### How to run s3fs, if applicable  [x] command line [] /etc/fstab  ``` s3fs -o compat_dir -o nomultipart bucket_name local_dir ``` #### s3fs syslog messages (`grep s3fs /var/log/syslog`, `journalctl | grep s3fs`, or `s3fs outputs`)  ``` Jul 30 13:44:28 ip-10-82-70-36 s3fs[604593]: s3fs version 1.92(4b3e715) : s3fs -o compat_dir -o nocopyapi hotspottx-idalab s3b_idalab Jul 30 13:44:28 ip-10-82-70-36 s3fs[604593]: Loaded mime information from /etc/mime.types Jul 30 13:44:28 ip-10-82-70-36 s3fs[604601]: s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function. Jul 30 13:44:28 ip-10-82-70-36 s3fs[604601]: init v1.92(commit:4b3e715) with OpenSSL, credential-library(built-in) Jul 30 15:19:08 ip-10-82-70-36 kernel: [109986.619304] s3fs[604601]: segfault at 7fda49dff910 ip 00007fda4c49659b sp 00007ffeb34dec30 error 4 in libc.so.6[7fda4c428000+195000] Jul 30 15:19:43 ip-10-82-70-36 s3fs[628180]: s3fs version 1.92(4b3e715) : s3fs -o compat_dir -o nomultipart hotspottx-idalab s3b_idalab Jul 30 15:19:43 ip-10-82-70-36 s3fs[628180]: Loaded mime information from /etc/mime.types Jul 30 15:19:43 ip-10-82-70-36 s3fs[628188]: s3fs_cred.cpp:VersionS3fsCredential(60): Check why built-in function was called, the external credential library must have VersionS3fsCredential function. Jul 30 15:19:43 ip-10-82-70-36 s3fs[628188]: init v1.92(commit:4b3e715) with OpenSSL, credential-library(built-in) Jul 30 16:22:04 ip-10-82-70-36 s3fs[629784]: s3fs version 1.92(4b3e715) : s3fs outputs Jul 30 16:22:04 ip-10-82-70-36 s3fs[629784]: Loaded mime information from /etc/mime.types Jul 30 16:22:04 ip-10-82-70-36 s3fs[629784]: s3fs: missing MOUNTPOINT argument. ``` ### Details about issue  Tried to cat file1 file2 > file3, and got the error of "File too large". Tried to generate file3 first and copy to bucket via rsync, got the same error. Initially I used `s3fs -o compat_dir bucket_name local_dir`, and played a few other options but didn't work. Any suggestions? Thanks!

kerem added the

need info

label

2026-03-04 01:51:38 +03:00

kerem commented

2026-03-04 01:51:39 +03:00

Author

Owner

@gaul commented on GitHub (Sep 8, 2023):

Let's be clear about which options you are using. The defaults use multi-part upload which generally enables > 5 GB objects. By specifying -o nomultipart this limits s3fs to single-part uploads. AWS S3 does not support > 5 GB single-part objects although some compatible object stores do. What are you doing specifically?

@gaul commented on GitHub (Sep 8, 2023): Let's be clear about which options you are using. The defaults use multi-part upload which generally enables > 5 GB objects. By specifying `-o nomultipart` this limits s3fs to single-part uploads. AWS S3 does not support > 5 GB single-part objects although some compatible object stores do. What are you doing specifically?

kerem commented

2026-03-04 01:51:39 +03:00

Author

Owner

@chunxuan-hs commented on GitHub (Sep 13, 2023):

The initial command I used is
s3fs -o compat_dir bucket_name local_dir,

However, it failed to copy a file larger than 100GB.

Then I searched on the internet, and it seems people suggested to use -o nomultipart. So I tried:
s3fs -o compat_dir -o nomultipart bucket_name local_dir,
but same error appeared.

@chunxuan-hs commented on GitHub (Sep 13, 2023): The initial command I used is `s3fs -o compat_dir bucket_name local_dir`, However, it failed to copy a file larger than 100GB. Then I searched on the internet, and it seems people suggested to use `-o nomultipart`. So I tried: `s3fs -o compat_dir -o nomultipart bucket_name local_dir`, but same error appeared.