starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #453] Big filesize upload. Zero size file in result #244

New issue

Closed

opened 2026-03-04 01:43:38 +03:00 by kerem · 8 comments

kerem commented

2026-03-04 01:43:38 +03:00

Owner

Originally created by @mobidevadmin on GitHub (Jul 22, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/453

Hello,

I'm trying to upload 42GB file to s3 via s3fs. Details are:

Ubuntu 14.04.1
V1.80(commit:b76fc35) with OpenSSL or V1.78 with OpenSSL
libcurl 7.35.0-1ubuntu2.1
s3fs -d -d -f -o f2 -o curldbg MyBucket:/ms1 /ms1backup/. Tried with use_cache option - same result
upload takes about 4 hours and according to network port statistics the upload fails on the very end of uploading or about that time as i see corresponding load on the LAN port.

Target directory sctructure is
ms1/
/daily
/weekly
/monthly
/yearly

There is file attached and upload command is
tar -cvf $TARGETDIR/$1/MyBackup_$DATE.tar.gz /storage/winbackup
It doesn't make sense, though, as i tried to upload with simple cp command and result is the same. Small files (at least up to 2.2 GB) uploaded fine. I have another backup with 34GB files which has been working fine for couple years now. It seems to me that it's something to do with the overall "session" time which the whole upload procedure takes.
s3fs_log.txt

Originally created by @mobidevadmin on GitHub (Jul 22, 2016). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/453 Hello, I'm trying to upload 42GB file to s3 via s3fs. Details are: - Ubuntu 14.04.1 - V1.80(commit:b76fc35) with OpenSSL or V1.78 with OpenSSL - libcurl 7.35.0-1ubuntu2.1 - s3fs -d -d -f -o f2 -o curldbg MyBucket:/ms1 /ms1backup/. Tried with use_cache option - same result - upload takes about 4 hours and according to network port statistics the upload fails on the very end of uploading or about that time as i see corresponding load on the LAN port. Target directory sctructure is ms1/ /daily /weekly /monthly /yearly There is file attached and upload command is tar -cvf $TARGETDIR/$1/MyBackup_$DATE.tar.gz /storage/winbackup It doesn't make sense, though, as i tried to upload with simple cp command and result is the same. Small files (at least up to 2.2 GB) uploaded fine. I have another backup with 34GB files which has been working fine for couple years now. It seems to me that it's something to do with the overall "session" time which the whole upload procedure takes. [s3fs_log.txt](https://github.com/s3fs-fuse/s3fs-fuse/files/377676/s3fs_log.txt)

kerem closed this issue

2026-03-04 01:43:38 +03:00

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 24, 2016):

@mobidevadmin
First of all, it is better that you should not specify the tar file output path under the mount point.
Because the tar file will be uploaded/downloaded the tar file at each time of this file changing, then it will take a lot of time.

And I think that you should specify about cache options(use_cache/max_stat_cache_size(and stat_cache_expire)/enable_noobj_cache/etc).
Please try to use these options.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Jul 24, 2016): @mobidevadmin First of all, it is better that you should not specify the tar file output path under the mount point. Because the tar file will be uploaded/downloaded the tar file at each time of this file changing, then it will take a lot of time. And I think that you should specify about cache options(use_cache/max_stat_cache_size(and stat_cache_expire)/enable_noobj_cache/etc). Please try to use these options. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner

@mobidevadmin commented on GitHub (Jul 24, 2016):

Hi ggtakec,

First of all, it is better that you should not specify the tar file output path under the mount point.
Because the tar file will be uploaded/downloaded the tar file at each time of this file changing, then it will take a lot of time

I tried to just copy 40+ GB file - same result

And I think that you should specify about cache options(use_cache/max_stat_cache_size(and stat_cache_expire)/enable_noobj_cache/etc).
Please try to use these options.

Could you please give a hint on parameters i should use for max_stat_cache_size(and stat_cache_expire)/enable_noobj_cache

Thanks

@mobidevadmin commented on GitHub (Jul 24, 2016): Hi ggtakec, > First of all, it is better that you should not specify the tar file output path under the mount point. > Because the tar file will be uploaded/downloaded the tar file at each time of this file changing, then it will take a lot of time I tried to just copy 40+ GB file - same result > And I think that you should specify about cache options(use_cache/max_stat_cache_size(and stat_cache_expire)/enable_noobj_cache/etc). > Please try to use these options. Could you please give a hint on parameters i should use for max_stat_cache_size(and stat_cache_expire)/enable_noobj_cache Thanks

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner

@ggtakec commented on GitHub (Jul 24, 2016):

@mobidevadmin
I think you sould use following(example) options.

-o max_stat_cache_size=1000 : this value is default, if you need more cache entry, you can increase it.
-o enable_noobj_cache : the cache for the object does not exist.

When s3fs uploads/downloads a file, s3fs checks the stats(permission) of the file and directories.
So these option sets the cache for those stats, these options are to reduce the number of accesses to the S3.

And if you need, you can use following options.
-o stat_cache_expire : if you do not enough memory, you should specify this option.
-o use_cache=/tmp : if you need.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Jul 24, 2016): @mobidevadmin I think you sould use following(example) options. -o max_stat_cache_size=1000 : this value is default, if you need more cache entry, you can increase it. -o enable_noobj_cache : the cache for the object does not exist. When s3fs uploads/downloads a file, s3fs checks the stats(permission) of the file and directories. So these option sets the cache for those stats, these options are to reduce the number of accesses to the S3. And if you need, you can use following options. -o stat_cache_expire : if you do not enough memory, you should specify this option. -o use_cache=/tmp : if you need. Thanks in advance for your assistance.

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner

@mobidevadmin commented on GitHub (Jul 24, 2016):

Thank you! I'll try and get back to you with results.

@mobidevadmin commented on GitHub (Jul 24, 2016): Thank you! I'll try and get back to you with results.

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner

@mobidevadmin commented on GitHub (Jul 24, 2016):

Here is an update,

i used these additional parameters. It failed.
-o max_stat_cache_size=2000 -o enable_noobj_cache -o use_cache=/tmp
s3fs_log2.txt

@mobidevadmin commented on GitHub (Jul 24, 2016): Here is an update, i used these additional parameters. It failed. -o max_stat_cache_size=2000 -o enable_noobj_cache -o use_cache=/tmp [s3fs_log2.txt](https://github.com/s3fs-fuse/s3fs-fuse/files/380445/s3fs_log2.txt)

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner

@mobidevadmin commented on GitHub (Jul 28, 2016):

Hey,

any ideas?

@mobidevadmin commented on GitHub (Jul 28, 2016): Hey, any ideas?

kerem commented

2026-03-04 01:43:39 +03:00

Author

Owner