starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00

[GH-ISSUE #570] Fall back to single-part upload when multi-part upload not supported #323

New issue

Closed

opened 2026-03-04 01:44:24 +03:00 by kerem · 6 comments

kerem commented

2026-03-04 01:44:24 +03:00

Owner

Originally created by @gaul on GitHub (Apr 29, 2017).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/570

s3fs should fall back from MPU when the endpoint emits HTTP 501 NotImplemented. This would eliminate the need for -o nomultipart for some users. This is similar to falling back from v4 to v2 authentication which s3fs already does.

Originally created by @gaul on GitHub (Apr 29, 2017). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/570 s3fs should fall back from MPU when the endpoint emits HTTP 501 NotImplemented. This would eliminate the need for `-o nomultipart` for some users. This is similar to falling back from v4 to v2 authentication which s3fs already does.

kerem closed this issue

2026-03-04 01:44:24 +03:00

kerem commented

2026-03-04 01:44:25 +03:00

Author

Owner

@sqlbot commented on GitHub (Apr 29, 2017):

It seems like there's a pretty strong argument to be made that users should simply configure their systems correctly, rather than the software engaging in such "helpful" fallback behavior like this and sig v2/v4... fallback behavior wastes time, resources, and sometimes money -- HTTP requests are cheap, but not free in any of these senses.

@sqlbot commented on GitHub (Apr 29, 2017): It seems like there's a pretty strong argument to be made that users should simply configure their systems correctly, rather than the software engaging in such "helpful" fallback behavior like this and sig v2/v4... fallback behavior wastes time, resources, and sometimes money -- HTTP requests are cheap, but not free in any of these senses.

kerem commented

2026-03-04 01:44:25 +03:00

Author

Owner

@gaul commented on GitHub (Apr 29, 2017):

Some systems do not support features like v4 signing or MPU so there is no way to properly configure it. Having worked with several and written a partially compatible S3 servers I do not believe all vendors will support all features. Currently users file issues when s3fs does not work which requires time for them and us to debug. We would only issue one failed request per instantiation of s3fs, not per operation, just like v2 fallback does. That is to say, s3fs would try the MPU path and if it fails, set nomultipart option so future calls issue single-part uploads.

@gaul commented on GitHub (Apr 29, 2017): Some systems do not support features like v4 signing or MPU so there is no way to properly configure it. Having worked with several and written a partially compatible S3 servers I do not believe all vendors will support all features. Currently users file issues when s3fs does not work which requires time for them and us to debug. We would only issue one failed request per instantiation of s3fs, not per operation, just like v2 fallback does. That is to say, s3fs would try the MPU path and if it fails, set `nomultipart` option so future calls issue single-part uploads.

kerem commented

2026-03-04 01:44:25 +03:00

Author

Owner

@sqlbot commented on GitHub (Apr 29, 2017):

Sorry, @andrewgaul... I wasn't clear. I meant having s3fs configured correctly -- not the storage platform. If the user wants to access a platform that only supports a subset of genuine S3, the user should be responsible for configuring the option to disable the incompatible feature or enable support for the workaround. Automatic, well-intentioned but hidden magic is bad, clear documention of how features work and their implementation is good.

Supporting the _$folder$ construct is another example of a "helpful" feature that should not be enabled by default (it was, the last I looked). It helps a fringe subset of users and wastes time and money for the majority.

@sqlbot commented on GitHub (Apr 29, 2017): Sorry, @andrewgaul... I wasn't clear. I meant having s3fs configured correctly -- not the storage platform. If the user wants to access a platform that only supports a subset of genuine S3, the user should be responsible for configuring the option to disable the incompatible feature or enable support for the workaround. Automatic, well-intentioned but hidden magic is bad, clear documention of how features work and their implementation is good. Supporting the `_$folder$` construct is another example of a "helpful" feature that should not be enabled by default (it was, the last I looked). It helps a fringe subset of users and wastes time and money for the majority.

kerem commented

2026-03-04 01:44:25 +03:00

Author

Owner

@ggtakec commented on GitHub (Apr 30, 2017):

I think that it may be good to be a fallback function from the MPU(the realization of this function is unexamined yet).
This is an advantage obtained by only judging an error once when multipart upload is executed.(a error should be 501 status)
However, I think that the nomultipart option should be retained.

Indeed, it should not be implemented if it is judged every time like _folder as @sqlbot said.
But in this case, I think whether it is sufficient to judge only once(only the first time of multipart upload).
It is unnecessary every time.
So I think it is good to implement "Fall back to single-part upload".

As an aside, support for the _folder etc are not good for performance/cost as you said, so I would like to consider escaping to options(TBD).

@ggtakec commented on GitHub (Apr 30, 2017): I think that it may be good to be a fallback function from the MPU(the realization of this function is unexamined yet). This is an advantage obtained by only judging an error once when multipart upload is executed.(a error should be 501 status) However, I think that the nomultipart option should be retained. Indeed, it should not be implemented if it is judged every time like _$folder$ as @sqlbot said. But in this case, I think whether it is sufficient to judge only once(only the first time of multipart upload). It is unnecessary every time. So I think it is good to implement "Fall back to single-part upload". As an aside, support for the _$folder$ etc are not good for performance/cost as you said, so I would like to consider escaping to options(TBD).

kerem commented

2026-03-04 01:44:25 +03:00

Author

Owner

@gaul commented on GitHub (Apr 30, 2017):

Agree that nomultipart option should be retained. I will open a separate issue for _$folder$ since I do not understand this well and it hurts performance.

@gaul commented on GitHub (Apr 30, 2017): Agree that `nomultipart` option should be retained. I will open a separate issue for `_$folder$` since I do not understand this well and it hurts performance.

kerem commented

2026-03-04 01:44:25 +03:00

Author

Owner

@gaul commented on GitHub (Apr 9, 2019):

Agree with @sqlbot; we should not make s3fs overly clever when an obvious workaround exists.

@gaul commented on GitHub (Apr 9, 2019): Agree with @sqlbot; we should not make s3fs overly clever when an obvious workaround exists.

kerem referenced this issue

2026-03-04 01:54:05 +03:00

[PR #323] [MERGED] Add goofys to references #1463