[GH-ISSUE #327] Large files are not uploading

kerem commented

2026-03-04 01:42:50 +03:00

Owner

Originally created by @postmaxin on GitHub (Jan 9, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/327

Using s3fs with rsync, any large files do not seem to upload. The symptom is really odd. rsync shows the file uploading at 25MB/s, which is impossible, I wouldn't expect to get more than 500-1000k/s out of my connection.

Rsync "finishes" uploading the file and moves onto the next one, however, when I go into s3, the file is zero bytes.

This only happens with really large files (eg; 10 gigabyte). Files that are only a few megabytes in size upload without a problem.

So there's really a few issues here:

Why does rsync think it is able to upload so fast?
Why isn't the upload completing?
Why aren't we even getting one byte out of the upload? I would expect a partial file to be left over (I'm using "--partial" and "--inplace" with rsync).

Originally created by @postmaxin on GitHub (Jan 9, 2016). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/327 Using s3fs with rsync, any large files do not seem to upload. The symptom is really odd. rsync shows the file uploading at 25MB/s, which is impossible, I wouldn't expect to get more than 500-1000k/s out of my connection. Rsync "finishes" uploading the file and moves onto the next one, however, when I go into s3, the file is zero bytes. This only happens with really large files (eg; 10 gigabyte). Files that are only a few megabytes in size upload without a problem. So there's really a few issues here: 1) Why does rsync think it is able to upload so fast? 2) Why isn't the upload completing? 3) Why aren't we even getting one byte out of the upload? I would expect a partial file to be left over (I'm using "--partial" and "--inplace" with rsync).

kerem closed this issue

2026-03-04 01:42:51 +03:00

kerem commented

2026-03-04 01:42:51 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 9, 2016):

@postmaxin
When s3fs copies a file, s3fs makes a file(object) as 0 byte at first. After that, s3fs copies a file to bucket.
And when you copy a file, s3fs makes it in temporary file in local. After copied a temporary file completely, s3fs uploads it to bucket.
Then it seems that s3fs failed to upload temporary file to your bucket.

rsync makes a file to temporary file(in local file system), so I think it does not include the time of upload for the rsync, or it does not take a long time because it failed to upload.
probably failed.
you should specify only "--inplace" option.(because of described above)

If you can, please run s3fs with "dbglevel" option(please see man page) and get some debug message about failure.
It helps us for solving this issue.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 9, 2016): @postmaxin When s3fs copies a file, s3fs makes a file(object) as 0 byte at first. After that, s3fs copies a file to bucket. And when you copy a file, s3fs makes it in temporary file in local. After copied a temporary file completely, s3fs uploads it to bucket. Then it seems that s3fs failed to upload temporary file to your bucket. 1) rsync makes a file to temporary file(in local file system), so I think it does not include the time of upload for the rsync, or it does not take a long time because it failed to upload. 2) probably failed. 3) you should specify only "--inplace" option.(because of described above) If you can, please run s3fs with "dbglevel" option(please see man page) and get some debug message about failure. It helps us for solving this issue. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:51 +03:00

Author

Owner

@postmaxin commented on GitHub (Jan 13, 2016):

--inplace implies --partial. I guess what I really should be doing is --append, but --append is only going to do me good if I can get at least one byte into the file on each run. ;)

I've generated a log at dbglevel, however, it is 3.2 gigabytes uncompressed. I'm attaching that syslog compressed, as well as a log of backup2l + rsync's output.

As you can see, rsync thinks it has successfully uploaded all of the files to the mounted s3fs partition. However, on the S3 side, majel.arnot.1.tar.bz2.gpg is zero bytes. Not only is the transfer to S3 said to be successful, but it happens at an impossibly fast speed.

This is where it gets really bad though: If you look at the rsync log, rsync thinks it's written additional files to s3fs... however, every file after majel.arnot.1.tar.bz2.gpg does not exist on S3! They also appear to go at rediculous speeds too (eg; majel.arnot.1001.tar.bz2.gpg at 225.02MB/s?!?!)

And then, finally, at the end, I'm told majel.arnot.1.tar.bz2.gpg had an error in ftruncate(), but none of the other missing files show an error.

Anyway, even zipped, the logs are 45MB, which is too big for upload here. I've uploaded them to S3 so you can download them: http://temp-public-20160113.s3-website-us-west-2.amazonaws.com/logfiles.zip -- let me know when you have them so I can delete them :)

Any idea how to fix this? If I could even get a few megabytes uploaded to the file, I could retry over and over with --append and eventually get the file up there... but with it always being zero bytes, I'm kind of stuck. :(

@postmaxin commented on GitHub (Jan 13, 2016): --inplace implies --partial. I guess what I really should be doing is --append, but --append is only going to do me good if I can get at least one byte into the file on each run. ;) I've generated a log at dbglevel, however, it is 3.2 gigabytes uncompressed. I'm attaching that syslog compressed, as well as a log of backup2l + rsync's output. As you can see, rsync thinks it has successfully uploaded all of the files to the mounted s3fs partition. However, on the S3 side, majel.arnot.1.tar.bz2.gpg is zero bytes. Not only is the transfer to S3 said to be successful, but it happens at an impossibly fast speed. This is where it gets really bad though: If you look at the rsync log, rsync thinks it's written _additional_ files to s3fs... however, every file after majel.arnot.1.tar.bz2.gpg does not exist on S3! They also appear to go at rediculous speeds too (eg; majel.arnot.1001.tar.bz2.gpg at 225.02MB/s?!?!) And then, finally, at the end, I'm told majel.arnot.1.tar.bz2.gpg had an error in ftruncate(), but none of the other missing files show an error. Anyway, even zipped, the logs are 45MB, which is too big for upload here. I've uploaded them to S3 so you can download them: http://temp-public-20160113.s3-website-us-west-2.amazonaws.com/logfiles.zip -- let me know when you have them so I can delete them :) Any idea how to fix this? If I could even get a few megabytes uploaded to the file, I could retry over and over with --append and eventually get the file up there... but with it always being zero bytes, I'm kind of stuck. :(

kerem commented

2026-03-04 01:42:52 +03:00

Author

Owner

@postmaxin commented on GitHub (Jan 13, 2016):

So the issue seems to be that I can't upload a file partially to S3. So there's a few things I can try:

Split the file into smaller chunks. It shouldn't be too hard to make backup2l do this
Play with s3fs options. Will boosting retries help? Are retries on the file level or multipart chunk level?

@postmaxin commented on GitHub (Jan 13, 2016): So the issue seems to be that I can't upload a file partially to S3. So there's a few things I can try: 1) Split the file into smaller chunks. It shouldn't be too hard to make backup2l do this 2) Play with s3fs options. Will boosting retries help? Are retries on the file level or multipart chunk level?

kerem commented

2026-03-04 01:42:52 +03:00

Author

Owner

@postmaxin commented on GitHub (Jan 14, 2016):

Okay, I think I found something relevant in the logs. s3fs is claiming that it cannot upload part 966 out of a 965-part upload?

Jan 14 09:03:57 majel s3fs[2295]: UploadMultipartPostRetryCallback(1075): Over r
etry count(10) limit(/majel.arnot/majel.arnot.1.tar.bz2.gpg:966).
Jan 14 09:03:57 majel s3fs[2295]: ParallelMultipartUploadRequest(1174): error occuered in multi request(errno=-5).
Jan 14 09:03:57 majel s3fs[2295]: [tpath=/majel.arnot/majel.arnot.1.tar.bz2.gpg][parts=966]
Jan 14 09:03:57 majel s3fs[2295]: CompleteMultipartPostRequest(2863): 965 file part is not finished uploading.
Jan 14 09:03:57 majel s3fs[2295]: [ent->file=/majel.arnot/majel.arnot.1.tar.bz2.gpg][ent->fd=7]
Jan 14 09:03:57 majel s3fs[2295]: [path=/majel.arnot/majel.arnot.1.tar.bz2.gpg][fd=7][refcnt=1]

@postmaxin commented on GitHub (Jan 14, 2016): Okay, I think I found something relevant in the logs. s3fs is claiming that it cannot upload part 966 out of a 965-part upload? Jan 14 09:03:57 majel s3fs[2295]: UploadMultipartPostRetryCallback(1075): Over r etry count(10) limit(/majel.arnot/majel.arnot.1.tar.bz2.gpg:966). Jan 14 09:03:57 majel s3fs[2295]: ParallelMultipartUploadRequest(1174): error occuered in multi request(errno=-5). Jan 14 09:03:57 majel s3fs[2295]: [tpath=/majel.arnot/majel.arnot.1.tar.bz2.gpg][parts=966] Jan 14 09:03:57 majel s3fs[2295]: CompleteMultipartPostRequest(2863): 965 file part is not finished uploading. Jan 14 09:03:57 majel s3fs[2295]: [ent->file=/majel.arnot/majel.arnot.1.tar.bz2.gpg][ent->fd=7] Jan 14 09:03:57 majel s3fs[2295]: [path=/majel.arnot/majel.arnot.1.tar.bz2.gpg][fd=7][refcnt=1]

kerem commented

2026-03-04 01:42:52 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 16, 2016):

I'm sorry for replying late.
And I could not load your log(zip) file because of 403 Forbidden.

But your lastest post said "Over retry count(10) limit(/majel.arnot/majel.arnot.1.tar.bz2.gpg:966).", so it means a part No.966 failed to upload despite 10 retried.
I hope if you can solve, it may be necessary to increase the number of retries option.
Please try it if you can.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 16, 2016): I'm sorry for replying late. And I could not load your log(zip) file because of 403 Forbidden. But your lastest post said "Over retry count(10) limit(/majel.arnot/majel.arnot.1.tar.bz2.gpg:966).", so it means a part No.966 failed to upload despite 10 retried. I hope if you can solve, it may be necessary to increase the number of retries option. Please try it if you can. Thanks in advance for your help.

kerem commented

2026-03-04 01:42:52 +03:00

Author

Owner

@postmaxin commented on GitHub (Jan 20, 2016):

Hi,

Sorry it's taken me so long to get back to you. I ended up doing two things: Boosted retries, and I am splitting my files into 4GB chunks before uploading, so that there's a greater chance of getting chunks into S3. Between these two, I now have a system where rsync can eventually get my incremental backups into S3.

So, in the end, there were four things that confused me. :)

First, I misunderstood how the S3 multipart uploads work.

Second, the presence of the zero-byte file in S3 after a failure threw me off. (Is the zero-byte file necessary?)

Third, the way that rsync handles failure -- it appears that when it hits a failure and is going to die, it still iterates over every file that's left in the list (even though it doesn't do anything with them). I've used rsync extensively in the past so I really should have remembered this!

Finally, the log message that I posted on January 14th. The way I read the log messages, it sounded like s3fs was saying that the file had 965 multiparts total, but it was failing to upload part #966 (which shouldn't exist if there was only 965 parts). I think maybe this could be made more clear by always saying how many parts there are (eg "File part 965 out of 4000 is not finished uploading").

Thank you for your help! And especially for s3fs. It's awesome. :)

@postmaxin commented on GitHub (Jan 20, 2016): Hi, Sorry it's taken me so long to get back to you. I ended up doing two things: Boosted retries, and I am splitting my files into 4GB chunks before uploading, so that there's a greater chance of getting chunks into S3. Between these two, I now have a system where rsync can eventually get my incremental backups into S3. So, in the end, there were four things that confused me. :) First, I misunderstood how the S3 multipart uploads work. Second, the presence of the zero-byte file in S3 after a failure threw me off. (Is the zero-byte file necessary?) Third, the way that rsync handles failure -- it appears that when it hits a failure and is going to die, it still iterates over every file that's left in the list (even though it doesn't _do_ anything with them). I've used rsync extensively in the past so I really should have remembered this! Finally, the log message that I posted on January 14th. The way I read the log messages, it sounded like s3fs was saying that the file had 965 multiparts _total_, but it was failing to upload part #966 (which shouldn't exist if there was only 965 parts). I think maybe this could be made more clear by always saying how many parts there are (eg "File part 965 out of 4000 is not finished uploading"). Thank you for your help! And especially for s3fs. It's awesome. :)

kerem referenced this issue

2026-03-04 01:53:39 +03:00

[PR #171] [MERGED] Support buckets with mixed-case names #1379

Rows
Columns

[GH-ISSUE #327] Large files are not uploading #171