[GH-ISSUE #534] S3 Sync implemented? #303

New issue

Closed

opened 2026-03-04 01:44:11 +03:00 by kerem · 4 comments

kerem commented

2026-03-04 01:44:11 +03:00

Owner

Originally created by @ylluminate on GitHub (Feb 15, 2017).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/534

Is Sync functionality implemented in s3fs-fuse as per this wherein synchronization is handled as it is explained there, namely:

only files that don’t exist at the destination in the same version are transferred by the s3cmd sync command. By default a md5 checksum and file size is compared. This is similar to a unix rsync command, with some exceptions outlined below.

Originally created by @ylluminate on GitHub (Feb 15, 2017). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/534 Is Sync functionality implemented in `s3fs-fuse` as per [this](http://s3tools.org/s3cmd-sync) wherein synchronization is handled as it is explained there, namely: > only files that don’t exist at the destination in the same version are transferred by the s3cmd sync command. By default a md5 checksum and file size is compared. This is similar to a unix rsync command, with some exceptions outlined below.

kerem closed this issue

2026-03-04 01:44:11 +03:00

kerem commented

2026-03-04 01:44:12 +03:00

Author

Owner

@gaul commented on GitHub (Feb 15, 2017):

No; s3cmd syncs files while s3fs writes files in real-time. With the former you work locally for reading and writing then initiate a sync while with the latter you work remotely and reads and writes propagate immediately.

@gaul commented on GitHub (Feb 15, 2017): No; s3cmd syncs files while s3fs writes files in real-time. With the former you work locally for reading and writing then initiate a sync while with the latter you work remotely and reads and writes propagate immediately.

kerem commented

2026-03-04 01:44:12 +03:00

Author

Owner

@ylluminate commented on GitHub (Feb 15, 2017):

So s3backer would be a union of both of these wherein operations are performed locally and then committed to S3, right?

What I'm ultimately stabbing at here is a solution for a service like Linode that doesn't offer a block storage option (yet) and need to immediately store large amounts of data in an efficient manner. I guess s3fs-fuse would match up to the need of a realtime block storage, but I worry about efficiency quite a bit on a disparate network like this where it has to go back and forth so much off (V)LAN for data...

@ylluminate commented on GitHub (Feb 15, 2017): So [`s3backer`](https://github.com/archiecobbs/s3backer) would be a union of both of these wherein operations are performed locally and then committed to S3, right? What I'm ultimately stabbing at here is a solution for a service like Linode that doesn't offer a block storage option (yet) and need to immediately store large amounts of data in an efficient manner. I guess `s3fs-fuse` would match up to the need of a realtime block storage, but I worry about efficiency quite a bit on a disparate network like this where it has to go back and forth so much off (V)LAN for data...

kerem commented

2026-03-04 01:44:12 +03:00

Author

Owner

@gaul commented on GitHub (Feb 16, 2017):

s3backer works well if you want a block device which asynchronously writes to S3. s3fs works at the file level and synchronizes writes when closing a file. This is a long shot but maybe you want to use something like jSCSI?

@gaul commented on GitHub (Feb 16, 2017): s3backer works well if you want a block device which asynchronously writes to S3. s3fs works at the file level and synchronizes writes when closing a file. This is a long shot but maybe you want to use something like [jSCSI](http://jscsi.org/)?

kerem commented

2026-03-04 01:44:12 +03:00

Author

Owner

@ylluminate commented on GitHub (Feb 17, 2017):

That is a very interesting suggestion @andrewgaul. It may be even more interesting in light of this:
http://kops.uni-konstanz.de/handle/123456789/26506

@ylluminate commented on GitHub (Feb 17, 2017): That is a very interesting suggestion @andrewgaul. It may be even more interesting in light of this: http://kops.uni-konstanz.de/handle/123456789/26506