[GH-ISSUE #1768] Large file upload failed with IO error, log shows `NoCacheLoadAndPost` #911

New issue

Open

opened 2026-03-04 01:49:50 +03:00 by kerem · 2 comments

kerem commented

2026-03-04 01:49:50 +03:00

Owner

Originally created by @dz902 on GitHub (Sep 27, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1768

Additional Information

The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all.
Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD

Version of s3fs being used (s3fs --version)

1.90

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2

Kernel information (uname -r)

4.14.243-185.433.amzn2.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

command result: cat /etc/os-release

s3fs command line used, if applicable

s3fs -o iam_role=auto -o -o multipart_size=1024 -o ensure_diskfree=5000 -o curldbg -o dbglevel=info -o logfile=/tmp/fuse4.log -f -d bucket /mnt/test

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages

This is what happened after last call of POST /dummyfile?uploads.

2021-09-27T08:25:37.317Z [INF]       curl.cpp:RequestPerform(2316): HTTP response code 200
2021-09-27T08:25:37.317Z [INF] fdcache_fdinfo.cpp:ClearUploadInfo(121): Implementation of cancellation process for multipart upload is awaited.
2021-09-27T08:25:37.317Z [INF]       fdcache_entity.cpp:NoCacheLoadAndPost(1061): [path=/dummyfile][physical_fd=12][offset=0][size=8590000128]
2021-09-27T08:25:37.318Z [ERR] fdcache_fdinfo.cpp:GetUploadId(145): Multipart Upload has not started yet.
2021-09-27T08:25:37.318Z [ERR] fdcache_entity.cpp:NoCacheLoadAndPost(1178): failed to multipart post(start=0, size=1073741824) for file(physical_fd=12).
2021-09-27T08:25:37.318Z [ERR] fdcache_entity.cpp:WriteMixMultipart(1993): failed to load uninitialized area and multipart uploading it(errno=-5)
2021-09-27T08:25:37.318Z [WAN] s3fs.cpp:s3fs_write(2363): failed to write file(/dummyfile). result=-5
2021-09-27T08:25:37.319Z [INF] s3fs.cpp:s3fs_flush(2398): [path=/dummyfile][pseudo_fd=2]
2021-09-27T08:25:37.319Z [INF]       fdcache_entity.cpp:RowFlush(1331): [tpath=][path=/dummyfile][pseudo_fd=2][physical_fd=12]
2021-09-27T08:25:37.319Z [INF]       fdcache_entity.cpp:RowFlushMixMultipart(1548): [tpath=][path=/dummyfile][pseudo_fd=2][physical_fd=12]
2021-09-27T08:25:37.319Z [ERR] fdcache_fdinfo.cpp:GetEtaglist(155): Multipart Upload has not started yet.
2021-09-27T08:25:37.319Z [ERR] fdcache_entity.cpp:NoCacheCompleteMultipartPost(1286): There is no upload id or etag list.
2021-09-27T08:25:37.319Z [ERR] fdcache_entity.cpp:RowFlushMixMultipart(1650): failed to complete(finish) multipart post for file(physical_fd=12).
2021-09-27T08:25:37.319Z [INF]       cache.cpp:DelStat(620): delete stat cache entry[path=/dummyfile]
2021-09-27T08:25:37.319Z [INF] s3fs.cpp:s3fs_release(2456): [path=/dummyfile][pseudo_fd=2]
2021-09-27T08:25:37.319Z [INF]       cache.cpp:DelStat(620): delete stat cache entry[path=/dummyfile]
2021-09-27T08:25:37.319Z [INF]       fdcache.cpp:GetFdEntity(473): [path=/dummyfile][pseudo_fd=2]

Details about issue

Root disk size: 16GB
Data disk size: 300GB
Dummy file on data disk: 250GB

Job fail with:

cp: error writing ‘/mnt/test/dummyfile’: Input/output error
cp: failed to extend ‘/mnt/test/dummyfile’: Input/output error
cp: failed to close ‘/mnt/test/dummyfile’: Input/output error

Also tried -o nomixupload, -o max_dirty_data=512, all result in 100% usage of root disk (or 100% minus ensure_diskfree), and their combinations to no avail.

It seems temporary data is never flush and keeps growing until tmp space is full and failed.

I have read numerous similar issues and thought this should have been fixed somehow but it is still there in v1.9.0.

There is a 8GB~14.5GB file as a result. Using different params may result in 0 length file or no file at all.

Originally created by @dz902 on GitHub (Sep 27, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1768 ### Additional Information _The following information is very important in order to help us to help you. Omission of the following details may delay your support request or receive no attention at all._ _Keep in mind that the commands we provide to retrieve information are oriented to GNU/Linux Distributions, so you could need to use others if you use s3fs on macOS or BSD_ #### Version of s3fs being used (s3fs --version) _1.90_ #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _2.9.2_ #### Kernel information (uname -r) _4.14.243-185.433.amzn2.x86_64_ #### GNU/Linux Distribution, if applicable (cat /etc/os-release) _command result: cat /etc/os-release_ #### s3fs command line used, if applicable ``` s3fs -o iam_role=auto -o -o multipart_size=1024 -o ensure_diskfree=5000 -o curldbg -o dbglevel=info -o logfile=/tmp/fuse4.log -f -d bucket /mnt/test ``` #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) _if you execute s3fs with dbglevel, curldbg option, you can get detail debug messages_ This is what happened after last call of `POST /dummyfile?uploads`. ``` 2021-09-27T08:25:37.317Z [INF] curl.cpp:RequestPerform(2316): HTTP response code 200 2021-09-27T08:25:37.317Z [INF] fdcache_fdinfo.cpp:ClearUploadInfo(121): Implementation of cancellation process for multipart upload is awaited. 2021-09-27T08:25:37.317Z [INF] fdcache_entity.cpp:NoCacheLoadAndPost(1061): [path=/dummyfile][physical_fd=12][offset=0][size=8590000128] 2021-09-27T08:25:37.318Z [ERR] fdcache_fdinfo.cpp:GetUploadId(145): Multipart Upload has not started yet. 2021-09-27T08:25:37.318Z [ERR] fdcache_entity.cpp:NoCacheLoadAndPost(1178): failed to multipart post(start=0, size=1073741824) for file(physical_fd=12). 2021-09-27T08:25:37.318Z [ERR] fdcache_entity.cpp:WriteMixMultipart(1993): failed to load uninitialized area and multipart uploading it(errno=-5) 2021-09-27T08:25:37.318Z [WAN] s3fs.cpp:s3fs_write(2363): failed to write file(/dummyfile). result=-5 2021-09-27T08:25:37.319Z [INF] s3fs.cpp:s3fs_flush(2398): [path=/dummyfile][pseudo_fd=2] 2021-09-27T08:25:37.319Z [INF] fdcache_entity.cpp:RowFlush(1331): [tpath=][path=/dummyfile][pseudo_fd=2][physical_fd=12] 2021-09-27T08:25:37.319Z [INF] fdcache_entity.cpp:RowFlushMixMultipart(1548): [tpath=][path=/dummyfile][pseudo_fd=2][physical_fd=12] 2021-09-27T08:25:37.319Z [ERR] fdcache_fdinfo.cpp:GetEtaglist(155): Multipart Upload has not started yet. 2021-09-27T08:25:37.319Z [ERR] fdcache_entity.cpp:NoCacheCompleteMultipartPost(1286): There is no upload id or etag list. 2021-09-27T08:25:37.319Z [ERR] fdcache_entity.cpp:RowFlushMixMultipart(1650): failed to complete(finish) multipart post for file(physical_fd=12). 2021-09-27T08:25:37.319Z [INF] cache.cpp:DelStat(620): delete stat cache entry[path=/dummyfile] 2021-09-27T08:25:37.319Z [INF] s3fs.cpp:s3fs_release(2456): [path=/dummyfile][pseudo_fd=2] 2021-09-27T08:25:37.319Z [INF] cache.cpp:DelStat(620): delete stat cache entry[path=/dummyfile] 2021-09-27T08:25:37.319Z [INF] fdcache.cpp:GetFdEntity(473): [path=/dummyfile][pseudo_fd=2] ``` ### Details about issue Root disk size: 16GB Data disk size: 300GB Dummy file on data disk: 250GB Job fail with: ``` cp: error writing ‘/mnt/test/dummyfile’: Input/output error cp: failed to extend ‘/mnt/test/dummyfile’: Input/output error cp: failed to close ‘/mnt/test/dummyfile’: Input/output error ``` Also tried `-o nomixupload`, `-o max_dirty_data=512`, all result in `100%` usage of root disk (or `100%` minus `ensure_diskfree`), and their combinations to no avail. It seems temporary data is never flush and keeps growing until tmp space is full and failed. I have read numerous similar issues and thought this should have been fixed somehow but it is still there in v1.9.0. There is a 8GB~14.5GB file as a result. Using different params may result in 0 length file or no file at all.

kerem commented

2026-03-04 01:49:51 +03:00

Author

Owner

@dz902 commented on GitHub (Sep 28, 2021):

Just tested that this happens on v1.9 and v1.89, but not on v1.87.

@dz902 commented on GitHub (Sep 28, 2021): Just tested that this happens on v1.9 and v1.89, but not on v1.87.

kerem commented

2026-03-04 01:49:51 +03:00

Author

Owner

@dz902 commented on GitHub (Sep 28, 2021):

Tried a second time (v1.87) with error:

cp: error writing ‘/mnt/test/dummy2’: Operation not permitted
cp: failed to extend ‘/mnt/test/dummy2’: Operation not permitted
cp: failed to close ‘/mnt/test/dummy2’: Operation not permitted

@dz902 commented on GitHub (Sep 28, 2021): Tried a second time (v1.87) with error: ``` cp: error writing ‘/mnt/test/dummy2’: Operation not permitted cp: failed to extend ‘/mnt/test/dummy2’: Operation not permitted cp: failed to close ‘/mnt/test/dummy2’: Operation not permitted ```