mirror of
https://github.com/s3fs-fuse/s3fs-fuse.git
synced 2026-04-25 21:35:58 +03:00
[GH-ISSUE #1527] Planning to use s3fs for our cloud ftp server. Maximum file size is 150GB #804
Labels
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/s3fs-fuse#804
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @SameerSawla on GitHub (Jan 20, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1527
Question
I am looking for the recommended way of using s3fs for a cloud ftp solution to cater files up to 150GB.
I see there are couple questions about the same but it is not really clear on what's the recommended way to use it.
I will be glad if someone could answer so it could help me and/or other folks are looking for the same question.
@gaul commented on GitHub (Jan 21, 2021):
s3fs can create large objects such as 150 GB, although there are temporary space and performance implications. s3fs 1.87 and earlier could only create files as large as the amount of temporary space of the local host. s3fs 1.88 added a feature to periodically flush uploads after a certain amount, controlled by
-o max_dirty_data(default="5120"). This will issue a multipart upload after reaching max_dirty_data, limiting the local storage required. s3fs uses themixuploadoptimization to issue server-side copies to "append" to the immutable S3 object. Unfortunately this causes extra IO on the S3 server so for a 150 GB object the server would do 5 + 10 + 15 + ... + 150 GB = 2,175 GB of IO. You can reduce this overhead by increasing themax_dirty_datavalue.@SameerSawla commented on GitHub (Jan 25, 2021):
Thanks for the response @gaul. I was curious about the 64GB limit that is mentioned in the documentation of s3fs-fuse. How does it impact the interaction with the larger files such as a 150GB one?
@gaul commented on GitHub (Jan 25, 2021):
There is no 64 GB limitation. Can you show me where this is documented? It should be corrected to 5 TB.
@gaul commented on GitHub (Feb 19, 2021):
Searching Google for "s3fs" "64GB" shows that both Fedora/Red Hat and Debian/Ubuntu document this incorrectly. I submitted a pull request for the former but how can we fix the latter?
@gaul commented on GitHub (Feb 19, 2021):
Submitted https://salsa.debian.org/debian/s3fs-fuse/-/merge_requests/3 for review.
@juliogonzalez commented on GitHub (Feb 19, 2021):
PR accepted for the Fedora/EPEL packages, I will submit it together with 1.89 if it happens soon, or separately if it takes too long to get the new version.
I verified that https://build.opensuse.org/package/view_file/filesystems/s3fs/s3fs.spec?expand=1 does not contain any reference to the maximum file size.
@gaul I am not 100% sure, but it could be you also need to add a changelog entry at https://salsa.debian.org/debian/s3fs-fuse/-/blob/debian/debian/changelog for 1.88-2 (on paper we need it for Fedora/EPEL, but I will handle it later on my own as I think I need to rebase a rebuild entry from Fedora)
@gaul commented on GitHub (Feb 22, 2021):
Both package descriptions PRs merged which should address the biggest source of confusion. I noticed several other inaccurate blogs posts but I don't think there is anything we can do about this.