starred/s3fs-fuse

Fork 0

mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00

[GH-ISSUE #402] S3 accelerate network #216

New issue

Closed

opened 2026-03-04 01:43:19 +03:00 by kerem · 6 comments

kerem commented

2026-03-04 01:43:19 +03:00

Owner

Originally created by @rossaffandy on GitHub (Apr 21, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/402

Hi guys, amazon just release accelerate network.
https://aws.amazon.com/blogs/aws/aws-storage-update-amazon-s3-transfer-acceleration-larger-snowballs-in-more-regions/

http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html

Any plan to support this feature? Currently, I'm just modifying s3fs.cpp and change endpoint from "us-east-1" to "accelerate"

Originally created by @rossaffandy on GitHub (Apr 21, 2016). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/402 Hi guys, amazon just release accelerate network. https://aws.amazon.com/blogs/aws/aws-storage-update-amazon-s3-transfer-acceleration-larger-snowballs-in-more-regions/ http://s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html Any plan to support this feature? Currently, I'm just modifying s3fs.cpp and change endpoint from "us-east-1" to "accelerate"

kerem

2026-03-04 01:43:19 +03:00

closed this issue
added the
question
label

kerem commented

2026-03-04 01:43:20 +03:00

Author

Owner

@jazzl0ver commented on GitHub (Apr 26, 2016):

Are you sure it's about endpoint? The docs you mentioned say we must change the url (s3fs.cpp):
std::string host = "http://s3.amazonaws.com";
to
std::string host = "http://s3-accelerate.amazonaws.com";

Also, I've compared copying a file to accelerated s3 bucket and to the general one and didn't find a difference.
Did you get better uploading rate in your tests?

@jazzl0ver commented on GitHub (Apr 26, 2016): Are you sure it's about endpoint? The docs you mentioned say we must change the url (s3fs.cpp): std::string host = "http://s3.amazonaws.com"; to std::string host = "http://s3-accelerate.amazonaws.com"; Also, I've compared copying a file to accelerated s3 bucket and to the general one and didn't find a difference. Did you get better uploading rate in your tests?

kerem commented

2026-03-04 01:43:20 +03:00

Author

Owner

@jsomontan commented on GitHub (May 6, 2016):

Any update on whether S3 acceleration gave you better performance? I'm think of creating a shared disk from S3 and sharing it out between ec2 instances, however, I 'd like to know if the transfer rates are an issue. Our systems are in the AWS Oregon region.

@jsomontan commented on GitHub (May 6, 2016): Any update on whether S3 acceleration gave you better performance? I'm think of creating a shared disk from S3 and sharing it out between ec2 instances, however, I 'd like to know if the transfer rates are an issue. Our systems are in the AWS Oregon region.

kerem commented

2026-03-04 01:43:20 +03:00

Author

Owner

@sqlbot commented on GitHub (May 6, 2016):

Note that if your instances and your bucket are in the same region, then transfer acceleration doesn't do anything. When that is the case, the acceleration feature should not even kick in -- DNS should send you to a normal S3 endpoint, and you shouldn't get billed the additional charge for that request. S3 is already capable of extremely fast transfers, and transfer acceleration doesn't change that -- it optimizes the connections for transfer over distance by using latency-based DNS to select a nearby (non-caching) edge proxy and uses the CloudFront managed transport network to connect back to S3. Within a region, your instance class's network throughput limitation, or the NAT instance's throughput (if not using an S3 VPC endpoint or NAT gateway) is usually the limiting factor. S3 itself is capable of extremely fast transfers.

@sqlbot commented on GitHub (May 6, 2016): Note that if your instances and your bucket are in the same region, then transfer acceleration doesn't do anything. When that is the case, [the acceleration feature should not even kick in -- DNS should send you to a normal S3 endpoint, and you shouldn't get billed the additional charge for that request.](http://aws.amazon.com/s3/faqs/#s3ta) S3 is already capable of extremely fast transfers, and transfer acceleration doesn't change that -- it optimizes the connections for transfer _over distance_ by using latency-based DNS to select a nearby (non-caching) edge proxy and uses the CloudFront managed transport network to connect back to S3. Within a region, your instance class's network throughput limitation, or the NAT instance's throughput (if not using an S3 VPC endpoint or NAT gateway) is usually the limiting factor. S3 itself is capable of extremely fast transfers.

kerem commented

2026-03-04 01:43:20 +03:00

Author

Owner

@jsomontan commented on GitHub (May 6, 2016):

Ok, so the limiting factor is the network of the instance. Thanks. But how does that compare to NFS mounts from a server/client versus if I create an S3FS filesystem on an S3 bucket? This is the goal I'm trying to achieve as I have this running in my environment. My NFS server is using EBS volumes.

@jsomontan commented on GitHub (May 6, 2016): Ok, so the limiting factor is the network of the instance. Thanks. But how does that compare to NFS mounts from a server/client versus if I create an S3FS filesystem on an S3 bucket? This is the goal I'm trying to achieve as I have this running in my environment. My NFS server is using EBS volumes.

kerem commented

2026-03-04 01:43:20 +03:00

Author

Owner

@jazzl0ver commented on GitHub (May 10, 2016):

@sqlbot, thanks for clarifying!
@jsomontan, the only issue with s3fs mount comparing to NFS mount is a file listing operation, which might take pretty long time in case you have a lot of files in a folder. transfer rates are pretty high and comfortable. anyway, i'd recommend you to make a simple test and compare speeds in your environment. from other side, s3fs (well, s3 actually) gives you such cool features like versioning and cross-region transfer, which general NFS service can't beat.

@jazzl0ver commented on GitHub (May 10, 2016): @sqlbot, thanks for clarifying! @jsomontan, the only issue with s3fs mount comparing to NFS mount is a file listing operation, which might take pretty long time in case you have a lot of files in a folder. transfer rates are pretty high and comfortable. anyway, i'd recommend you to make a simple test and compare speeds in your environment. from other side, s3fs (well, s3 actually) gives you such cool features like versioning and cross-region transfer, which general NFS service can't beat.

kerem commented

2026-03-04 01:43:20 +03:00

Author

Owner

@ggtakec commented on GitHub (May 14, 2016):

@rossaffandy I'm sorry for my late reply.
@jazzl0ver, @jsomontan and @sqlbot Thank you for your help.

First, if you need to change the end point, you can use "-o url=" option for s3fs.
And if you change the end point to "http://s3-accelerate.amazonaws.com" by url option, s3fs will be able to work the same as before the change.

But the response code s3fs has access to s3-accelerate.amazonaws.com is 307(Temporary Redirect).
s3fs sets the option for libcurl as CURLOPT_FOLLOWLOCATION(true), thus it will follow automatically location inside libcurl.
Finally, s3fs will be to resend the request to your bucket region.

With the above operation, you will be changed to s3-accelerate.amazonaws.com, I think you can not be obtained the anything good results.

And I think that description of accelerate network is as @sqlbot said.

Last, s3fs is slower than NFS because s3fs is using HTTP(S) behind FUSE(IPC), and thanks @jazzl0ver.

I'm closing this issue, if there is a issue to continue, please reopen this or post new one.

Thanks all

@ggtakec commented on GitHub (May 14, 2016): @rossaffandy I'm sorry for my late reply. @jazzl0ver, @jsomontan and @sqlbot Thank you for your help. First, if you need to change the end point, you can use "-o url=" option for s3fs. And if you change the end point to "http://s3-accelerate.amazonaws.com" by url option, s3fs will be able to work the same as before the change. But the response code s3fs has access to s3-accelerate.amazonaws.com is 307(Temporary Redirect). s3fs sets the option for libcurl as CURLOPT_FOLLOWLOCATION(true), thus it will follow automatically location inside libcurl. Finally, s3fs will be to resend the request to your bucket region. With the above operation, you will be changed to s3-accelerate.amazonaws.com, I think you can not be obtained the anything good results. And I think that description of accelerate network is as @sqlbot said. Last, s3fs is slower than NFS because s3fs is using HTTP(S) behind FUSE(IPC), and thanks @jazzl0ver. I'm closing this issue, if there is a issue to continue, please reopen this or post new one. Thanks all