starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00

[GH-ISSUE #312] Support for deep directory rename #161

New issue

Closed

opened 2026-03-04 01:42:45 +03:00 by kerem · 5 comments

kerem commented

2026-03-04 01:42:45 +03:00

Owner

Originally created by @saurabhdas on GitHub (Dec 4, 2015).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/312

Hello,

Thank you for this wonderful project!

The FAQ says:

s3fs does not support deep directory rename and doesn't check for it either.

Is there any plan to support feature this in the future?

Originally created by @saurabhdas on GitHub (Dec 4, 2015). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/312 Hello, Thank you for this wonderful project! The FAQ says: > s3fs does not support deep directory rename and doesn't check for it either. Is there any plan to support feature this in the future?

kerem closed this issue

2026-03-04 01:42:46 +03:00

kerem commented

2026-03-04 01:42:47 +03:00

Author

Owner

@sqlbot commented on GitHub (Dec 4, 2015):

I can't speak for the developers, but the core problem, the way I see it, is that the limitation is primarily one of the underlying service -- Amazon S3 -- which, itself, doesn't support renaming a "directory" at all. S3 isn't actually hierarchical, and files are not "contained" in folders in S3 -- the folders appear essentially as an artifact of the / delimiter being used in the file path... so it's not that S3 won't let you rename the "container" (folder) -- it's that there's no actual container to rename.

To rename an entire hierarchy of files in S3 requires sending put/copy (to the new path) requests and delete (from the old path) requests for every single object in the chain. If there are a million files, that's roughly 2 million API requests (fewer with batch deletes, but more when the files are large, because 5GB files require multipart operations).

@sqlbot commented on GitHub (Dec 4, 2015): I can't speak for the developers, but the core problem, the way I see it, is that the limitation is primarily one of the underlying service -- Amazon S3 -- which, itself, doesn't support renaming a "directory" at all. S3 isn't actually hierarchical, and files are not "contained" in folders in S3 -- the folders appear essentially as an artifact of the `/` delimiter being used in the file path... so it's not that S3 won't let you rename the "container" (folder) -- it's that there's no actual container to rename. To rename an entire hierarchy of files in S3 requires sending put/copy (to the new path) requests and delete (from the old path) requests for every single object in the chain. If there are a million files, that's roughly 2 million API requests (fewer with batch deletes, but more when the files are large, because 5GB files require multipart operations).

kerem commented

2026-03-04 01:42:47 +03:00

Author

Owner

@RobbKistler commented on GitHub (Dec 4, 2015):

I think the FAQ is out of date. s3fs-fuse has code that will do the deep rename, but it has to recursively rename all of the objects. It might not be fast, but it should work. It uses server-side copies for renames, so the objects don't necessarily have to make a round-trip.

@RobbKistler commented on GitHub (Dec 4, 2015): I think the FAQ is out of date. s3fs-fuse has code that will do the deep rename, but it has to recursively rename all of the objects. It might not be fast, but it should work. It uses server-side copies for renames, so the objects don't necessarily have to make a round-trip.

kerem commented

2026-03-04 01:42:47 +03:00

Author

Owner

@saurabhdas commented on GitHub (Dec 5, 2015):

Thanks for clearing that up @RobbKistler . A recursive rename of all objects is a workable solution in this case - slow is fine as long as files don't inadvertently go missing.

I'll close this issue. Perhaps the FAQ should be updated too.

Thanks again for all the good work here :)

@saurabhdas commented on GitHub (Dec 5, 2015): Thanks for clearing that up @RobbKistler . A recursive rename of all objects is a workable solution in this case - slow is fine as long as files don't inadvertently go missing. I'll close this issue. Perhaps the FAQ should be updated too. Thanks again for all the good work here :)

kerem commented

2026-03-04 01:42:47 +03:00

Author

Owner

@RobbKistler commented on GitHub (Dec 5, 2015):

@ggtakec I took the liberty of updating the FAQ. I removed the Q/A about directory rename not working. It seems to work fine. Is this OK?

@RobbKistler commented on GitHub (Dec 5, 2015): @ggtakec I took the liberty of updating the FAQ. I removed the Q/A about directory rename not working. It seems to work fine. Is this OK?

kerem commented

2026-03-04 01:42:47 +03:00

Author

Owner

@ggtakec commented on GitHub (Dec 20, 2015):

@RobbKistler thanks for updating FAQ.
The renaming dir should be running normally. FAQ has remained old.
Regards,

@ggtakec commented on GitHub (Dec 20, 2015): @RobbKistler thanks for updating FAQ. The renaming dir should be running normally. FAQ has remained old. Regards,

kerem referenced this issue

2026-03-04 01:53:38 +03:00

[PR #164] [MERGED] send the correct Host header when using -o url #1374