[GH-ISSUE #21] s3fs-fuse doesn't allow rsync delta transfer. #15

New issue

Closed

opened 2026-03-04 01:41:10 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:41:10 +03:00

Owner

Originally created by @CLloyd24 on GitHub (Mar 5, 2014).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/21

It is my understanding that s3fs-fuse is supposed to support rsync but it does not appear to support its most important/unique feature which is delta encoding.

Changing the id3 tags on some MP3s and rerunning rsync:

~$ rsync -avsh --delete --progress --no-whole-file --inplace ~/Random-Songs ~/DreamObjects

rsync runs for about as long as I would expect the 50 MBs to take to upload at ~2 Mbps. rsync reports the following:

sent 114.96K bytes  received 121.78K bytes  1.07K bytes/sec
total size is 52.16M  speedup is 220.30

Am I doing anything incorrectly or is this simply a limitation of rsync/S3/s3fs-fuse?

Thank you

The full output is:

~$ rsync -avsh --delete --progress --no-whole-file --inplace ~/Random-Songs ~/DreamObjects
sending incremental file list
Random-Songs/Aztec Camera -  Just Like Gold.mp3
       3.15M 100%   75.06MB/s    0:00:00 (xfer#1, to-check=11/13)
Random-Songs/Aztec Camera -  Lost Outside The Tunnel.mp3
       3.30M 100%   69.76MB/s    0:00:00 (xfer#2, to-check=10/13)
Random-Songs/Aztec Camera -  We Could Send Letters.mp3
       4.75M 100%   43.97MB/s    0:00:00 (xfer#3, to-check=9/13)
Random-Songs/Aztec Camera - Just Like Gold (7')-5jtzu8RdicQ.mp4
       7.26M 100%   40.26MB/s    0:00:00 (xfer#4, to-check=8/13)
Random-Songs/Aztec Camera - Lost Outside the Tunnel-909-YjBIjAY.mp4
       6.16M 100%  514.79kB/s    0:00:11 (xfer#5, to-check=7/13)
Random-Songs/Aztec Camera - Mattress Of Wire.mp3
       2.64M 100%    1.11MB/s    0:00:02 (xfer#6, to-check=6/13)
Random-Songs/Aztec Camera - Mattress of Wire-N2ozbgxQyJE.mp4
       5.07M 100%  227.41kB/s    0:00:21 (xfer#7, to-check=5/13)
Random-Songs/Aztec Camera - We Could Send Letters (7' Version)-wO_6Y4oT7UQ.mp4
      10.35M 100%  331.93kB/s    0:00:30 (xfer#8, to-check=4/13)
Random-Songs/Seven Saturdays - Eleven Eleven.mp3
       4.73M 100%    2.81MB/s    0:00:01 (xfer#9, to-check=3/13)

sent 114.96K bytes  received 121.78K bytes  1.07K bytes/sec
total size is 52.16M  speedup is 220.30

Originally created by @CLloyd24 on GitHub (Mar 5, 2014). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/21 It is my understanding that s3fs-fuse is supposed to support rsync but it does not appear to support its most important/unique feature which is delta encoding. Changing the id3 tags on some MP3s and rerunning rsync: ``` ~$ rsync -avsh --delete --progress --no-whole-file --inplace ~/Random-Songs ~/DreamObjects ``` rsync runs for about as long as I would expect the 50 MBs to take to upload at ~2 Mbps. rsync reports the following: ``` sent 114.96K bytes received 121.78K bytes 1.07K bytes/sec total size is 52.16M speedup is 220.30 ``` Am I doing anything incorrectly or is this simply a limitation of rsync/S3/s3fs-fuse? Thank you The full output is: ``` ~$ rsync -avsh --delete --progress --no-whole-file --inplace ~/Random-Songs ~/DreamObjects sending incremental file list Random-Songs/Aztec Camera - Just Like Gold.mp3 3.15M 100% 75.06MB/s 0:00:00 (xfer#1, to-check=11/13) Random-Songs/Aztec Camera - Lost Outside The Tunnel.mp3 3.30M 100% 69.76MB/s 0:00:00 (xfer#2, to-check=10/13) Random-Songs/Aztec Camera - We Could Send Letters.mp3 4.75M 100% 43.97MB/s 0:00:00 (xfer#3, to-check=9/13) Random-Songs/Aztec Camera - Just Like Gold (7')-5jtzu8RdicQ.mp4 7.26M 100% 40.26MB/s 0:00:00 (xfer#4, to-check=8/13) Random-Songs/Aztec Camera - Lost Outside the Tunnel-909-YjBIjAY.mp4 6.16M 100% 514.79kB/s 0:00:11 (xfer#5, to-check=7/13) Random-Songs/Aztec Camera - Mattress Of Wire.mp3 2.64M 100% 1.11MB/s 0:00:02 (xfer#6, to-check=6/13) Random-Songs/Aztec Camera - Mattress of Wire-N2ozbgxQyJE.mp4 5.07M 100% 227.41kB/s 0:00:21 (xfer#7, to-check=5/13) Random-Songs/Aztec Camera - We Could Send Letters (7' Version)-wO_6Y4oT7UQ.mp4 10.35M 100% 331.93kB/s 0:00:30 (xfer#8, to-check=4/13) Random-Songs/Seven Saturdays - Eleven Eleven.mp3 4.73M 100% 2.81MB/s 0:00:01 (xfer#9, to-check=3/13) sent 114.96K bytes received 121.78K bytes 1.07K bytes/sec total size is 52.16M speedup is 220.30 ```

kerem closed this issue

2026-03-04 01:41:11 +03:00

kerem commented

2026-03-04 01:41:12 +03:00

Author

Owner

@ggtakec commented on GitHub (Jan 17, 2016):

I'm sorry for that this issue had been left at a long period of time.

I think that you should not specify "--no-whole-file" option for rsync because S3 and s3fs can not update the partial file.
If you have this problem yet, please try to use latest code.
And if you found a problem, please post new issue.

Thanks in advance for your help.

@ggtakec commented on GitHub (Jan 17, 2016): I'm sorry for that this issue had been left at a long period of time. I think that you should not specify "--no-whole-file" option for rsync because S3 and s3fs can not update the partial file. If you have this problem yet, please try to use latest code. And if you found a problem, please post new issue. Thanks in advance for your help.

kerem referenced this issue

2026-03-04 01:43:14 +03:00

[GH-ISSUE #384] All files mounted from S3 are corrupted #202

kerem referenced this issue

2026-03-04 01:47:21 +03:00

[GH-ISSUE #1191] s3fs crash during cache read #630

kerem referenced this issue

2026-03-04 01:54:13 +03:00

[PR #394] [MERGED] Revert "Fixed a bug about writing sparsed file - #375" #1490

kerem referenced this issue

2026-03-04 01:54:14 +03:00

[PR #395] [MERGED] Fixed writing sparsed file - #375,#379,#394 #1492