[PR #1599] [MERGED] Don't ignore nomultipart when storage is low #2064

New issue

Closed

opened 2026-03-04 02:03:30 +03:00 by kerem · 0 comments

kerem commented

2026-03-04 02:03:30 +03:00

Owner

📋 Pull Request Information

Original PR: https://github.com/s3fs-fuse/s3fs-fuse/pull/1599
Author: @CarstenGrohmann
Created: 3/9/2021
Status: ✅ Merged
Merged: 3/9/2021
Merged by: @gaul

Base: master ← Head: nomultipart_on_space_shortage

📝 Commits (1)

739f499 Don't ignore nomultipart when storage is low

📊 Changes

1 file changed (+4 additions, -0 deletions)

View changed files

📝 src/fdcache_entity.cpp (+4 -0)

📄 Description

When the temporary storage filled up, the old implementation uploaded all data with multipart uploads, even if "nomultipart" was set.

The new implementation emits a warning and returns -ENOSPC instead.

Fixes #1595

Exampe:

$ cp -p 3000M /s3/mybucket/
cp: error writing "/s3/mybucket/3000M": No space left on device
cp: failed to extend "/s3/mybucket/3000M": No space left on device

New syslog output w/ warning:

2021-03-08T19:12:16.609323+01:00 mysrv <user.warning> s3fs[4842]:fdcache_entity.cpp:Write(1485): Not enough local storage to cache write request: [path=/3000M][fd=6][offset=1835008][size=65536]
2021-03-08T19:12:16.609522+01:00 mysrv <user.warning> s3fs[4842]:s3fs.cpp:s3fs_write(2310): failed to write file(/3000M). result=-28

New debug output w/ warning:

unique: 35, opcode: WRITE (16), nodeid: 2, insize: 65616, pid: 3759
write[5] 65536 bytes to 1835008 flags: 0x8001
2021-03-08T19:23:37.509Z [DBG] s3fs.cpp:s3fs_write(2298): [path=/3000M][size=65536][offset=1835008][fd=5]
2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:ExistOpen(526): [path=/3000M][fd=5][ignore_existfd=false]
2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:Open(446): [path=/3000M][size=-1][time=-1]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Dup(246): [path=/3000M][fd=5][refcnt=2]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Open(315): [path=/3000M][fd=5][size=-1][time=-1]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Dup(246): [path=/3000M][fd=5][refcnt=3]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Close(200): [path=/3000M][fd=5][refcnt=2]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Write(1439): [path=/3000M][fd=5][offset=1835008][size=65536]
2021-03-08T19:23:37.509Z [WAN] fdcache_entity.cpp:Write(1485): Not enough local storage to cache write request: [path=/3000M][fd=5][offset=1835008][size=65536]
2021-03-08T19:23:37.509Z [WAN] s3fs.cpp:s3fs_write(2310): failed to write file(/3000M). result=-28
2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:Close(593): [ent->file=/3000M][ent->fd=5]
2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Close(200): [path=/3000M][fd=5][refcnt=1]
   unique: 35, error: -28 (No space left on device), outsize: 16

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/s3fs-fuse/s3fs-fuse/pull/1599 **Author:** [@CarstenGrohmann](https://github.com/CarstenGrohmann) **Created:** 3/9/2021 **Status:** ✅ Merged **Merged:** 3/9/2021 **Merged by:** [@gaul](https://github.com/gaul) **Base:** `master` ← **Head:** `nomultipart_on_space_shortage` --- ### 📝 Commits (1) - [`739f499`](https://github.com/s3fs-fuse/s3fs-fuse/commit/739f499158c3823a0365e7f3acdd4b10082c6987) Don't ignore nomultipart when storage is low ### 📊 Changes **1 file changed** (+4 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `src/fdcache_entity.cpp` (+4 -0) </details> ### 📄 Description When the temporary storage filled up, the old implementation uploaded all data with multipart uploads, even if "nomultipart" was set. The new implementation emits a warning and returns -ENOSPC instead. Fixes #1595 **Exampe:** ``` $ cp -p 3000M /s3/mybucket/ cp: error writing "/s3/mybucket/3000M": No space left on device cp: failed to extend "/s3/mybucket/3000M": No space left on device ``` **New syslog output w/ warning:** ``` 2021-03-08T19:12:16.609323+01:00 mysrv <user.warning> s3fs[4842]:fdcache_entity.cpp:Write(1485): Not enough local storage to cache write request: [path=/3000M][fd=6][offset=1835008][size=65536] 2021-03-08T19:12:16.609522+01:00 mysrv <user.warning> s3fs[4842]:s3fs.cpp:s3fs_write(2310): failed to write file(/3000M). result=-28 ``` **New debug output w/ warning:** ``` unique: 35, opcode: WRITE (16), nodeid: 2, insize: 65616, pid: 3759 write[5] 65536 bytes to 1835008 flags: 0x8001 2021-03-08T19:23:37.509Z [DBG] s3fs.cpp:s3fs_write(2298): [path=/3000M][size=65536][offset=1835008][fd=5] 2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:ExistOpen(526): [path=/3000M][fd=5][ignore_existfd=false] 2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:Open(446): [path=/3000M][size=-1][time=-1] 2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Dup(246): [path=/3000M][fd=5][refcnt=2] 2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Open(315): [path=/3000M][fd=5][size=-1][time=-1] 2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Dup(246): [path=/3000M][fd=5][refcnt=3] 2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Close(200): [path=/3000M][fd=5][refcnt=2] 2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Write(1439): [path=/3000M][fd=5][offset=1835008][size=65536] 2021-03-08T19:23:37.509Z [WAN] fdcache_entity.cpp:Write(1485): Not enough local storage to cache write request: [path=/3000M][fd=5][offset=1835008][size=65536] 2021-03-08T19:23:37.509Z [WAN] s3fs.cpp:s3fs_write(2310): failed to write file(/3000M). result=-28 2021-03-08T19:23:37.509Z [DBG] fdcache.cpp:Close(593): [ent->file=/3000M][ent->fd=5] 2021-03-08T19:23:37.509Z [DBG] fdcache_entity.cpp:Close(200): [path=/3000M][fd=5][refcnt=1] unique: 35, error: -28 (No space left on device), outsize: 16 ``` --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>