[GH-ISSUE #435 ] Some files partially contain NUL content after uploading via S3FS (file size matches) #234

Author

Owner

@ClemensSchneider commented on GitHub (Jun 15, 2016):

After some more investigation I found some files that do partially contain correct content but then have a block of NUL content following with correct content afterwards again... How could this error-pattern occur? It leaves me with a bad feeling about the consitency of the data being written with S3FS.

@ClemensSchneider commented on GitHub (Jun 15, 2016): After some more investigation I found some files that do partially contain correct content but then have a block of NUL content following with correct content afterwards again... How could this error-pattern occur? It leaves me with a bad feeling about the consitency of the data being written with S3FS.

kerem commented

Author

Owner

@gaul commented on GitHub (Jun 15, 2016):

@ClemensSchneider Can you verify whether the file failed to upload or download correctly using another tool? s3fs has had other bugs around network errors which we have fixed but another alarms me as well. I wrote a tool Chaos HTTP Proxy which helped diagnose these and encourage you to test with it as well.

@gaul commented on GitHub (Jun 15, 2016): @ClemensSchneider Can you verify whether the file failed to upload or download correctly using another tool? s3fs has had other bugs around network errors which we have fixed but another alarms me as well. I wrote a tool [Chaos HTTP Proxy](https://github.com/bouncestorage/chaos-http-proxy) which helped diagnose these and encourage you to test with it as well.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jun 16, 2016):

@andrewgaul I verified that the files failed to upload correctly as I downloaded them using both S3-Browser and Dragondisk. In our case these files are FLAC files we are syncing to S3 using BitTorrent Sync. We noticed the data-corruption as our tool that is running in the cloud and is picking up changed metadata in the files complained about invalid FLAC-metadata.
It seems that an initial sync of the files worked correctly and following modifications of the metadata or audio-content leave some of the FLACs corrupted.

The bucket is mounted via /etc/fstab:
bucketname /mnt/s3 fuse.s3fs _netdev,url=http://s3.amazonaws.com,enable_noobj_cache,max_stat_cache_size=50000,stat_cache_expire=300,retries=10,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000 0 0

@ClemensSchneider commented on GitHub (Jun 16, 2016): @andrewgaul I verified that the files failed to upload correctly as I downloaded them using both S3-Browser and Dragondisk. In our case these files are FLAC files we are syncing to S3 using BitTorrent Sync. We noticed the data-corruption as our tool that is running in the cloud and is picking up changed metadata in the files complained about invalid FLAC-metadata. It seems that an initial sync of the files worked correctly and following modifications of the metadata or audio-content leave some of the FLACs corrupted. The bucket is mounted via /etc/fstab: `bucketname /mnt/s3 fuse.s3fs _netdev,url=http://s3.amazonaws.com,enable_noobj_cache,max_stat_cache_size=50000,stat_cache_expire=300,retries=10,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000 0 0`

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jun 24, 2016):

@ggtakec @andrewgaul Seems like writing only parts of a file leaves the file corrupt. Should S3FS be able to handle this case? I would expect S3FS to correctly apply the changes and then re-upload the whole file to S3.

I tried to enable MD5 checksum tests and to disable multipart uploads without any success:

bucketname /mnt/s3 fuse.s3fs _netdev,url=http://s3.amazonaws.com,enable_noobj_cache,max_stat_cache_size=50000,stat_cache_expire=300,retries=10,multireq_max=50,nomultipart,enable_content_md5,allow_other,rw,nonempty,uid=1000,gid=1000 0 0

Is there anything else I could do to fix this issue?

@ClemensSchneider commented on GitHub (Jun 24, 2016): @ggtakec @andrewgaul Seems like writing only parts of a file leaves the file corrupt. Should S3FS be able to handle this case? I would expect S3FS to correctly apply the changes and then re-upload the whole file to S3. I tried to enable MD5 checksum tests and to disable multipart uploads without any success: `bucketname /mnt/s3 fuse.s3fs _netdev,url=http://s3.amazonaws.com,enable_noobj_cache,max_stat_cache_size=50000,stat_cache_expire=300,retries=10,multireq_max=50,nomultipart,enable_content_md5,allow_other,rw,nonempty,uid=1000,gid=1000 0 0` Is there anything else I could do to fix this issue?

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jun 24, 2016):

I re-tested using S3FS v1.80, v1.79 and v1.78 and could reproduce this problematic behavior pretty reliably with both 1.80 and 1.79, but not (yet?) with 1.78 so it must have sneaked in between 1.78 and 1.79.

@ClemensSchneider commented on GitHub (Jun 24, 2016): I re-tested using S3FS v1.80, v1.79 and v1.78 and could reproduce this problematic behavior pretty reliably with both 1.80 and 1.79, but not (yet?) with 1.78 so it must have sneaked in between 1.78 and 1.79.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jul 11, 2016):

@ggtakec Is there anything I can provide to help diagnose this issue? Seems that the file corruption has not entirely gone after switching back to 1.78.
Could you please let me know if S3FS should support random-writes to files? It's clear to me, that the whole file has to be uploaded afterwards, but is the usecase itself supported?

@ClemensSchneider commented on GitHub (Jul 11, 2016): @ggtakec Is there anything I can provide to help diagnose this issue? Seems that the file corruption has not entirely gone after switching back to 1.78. Could you please let me know if S3FS should support random-writes to files? It's clear to me, that the whole file has to be uploaded afterwards, but is the usecase itself supported?

kerem commented

Author

Owner

@ggtakec commented on GitHub (Jul 18, 2016):

@ClemensSchneider I'm sorry for my late reply.

s3fs creates a local temporary file before uploading the file, and s3fs uploads it by S3 API.
And when you upload the file which is modified partially, s3fs downloads the other parts of the file from s3 before uploading.

This issue seems that s3fs failed downloading other parts before downloading.
Is all the contents of the problem files filled NULL charactor?
Or, did the problem files have only modifying parts?

I'm fixing #428(#444) which is a bug related to stats cache file and temporary file.
(If you can, please use the latest codes in master branch.)

And I think that we should know about your free disk space.
Does your machine have enough disk free space?
If you do not have enough disk space, you should use ensure_diskfree option.
It will affect the operation of upload and download, and the temporary file.

Last, if you can, please run s3fs on forground(-f option), and run with debug option(-dbglevel=XXX).
(If you use the debug option, please note that a large amount of debug messages are output.)
I expect that there is information in the file that failed in the debug message.

I'm sorry I can not be resolved immediately.
Thanks in advance for your help.

@ggtakec commented on GitHub (Jul 18, 2016): @ClemensSchneider I'm sorry for my late reply. s3fs creates a local temporary file before uploading the file, and s3fs uploads it by S3 API. And when you upload the file which is modified partially, s3fs downloads the other parts of the file from s3 before uploading. This issue seems that s3fs failed downloading other parts before downloading. Is all the contents of the problem files filled NULL charactor? Or, did the problem files have only modifying parts? I'm fixing #428(#444) which is a bug related to stats cache file and temporary file. (If you can, please use the latest codes in master branch.) And I think that we should know about your free disk space. Does your machine have enough disk free space? If you do not have enough disk space, you should use ensure_diskfree option. It will affect the operation of upload and download, and the temporary file. Last, if you can, please run s3fs on forground(-f option), and run with debug option(-dbglevel=XXX). (If you use the debug option, please note that a large amount of debug messages are output.) I expect that there is information in the file that failed in the debug message. I'm sorry I can not be resolved immediately. Thanks in advance for your help.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jul 19, 2016):

@ggtakec Thanks for your reply, I will try and answer as detailled as possible:

This issue seems that s3fs failed downloading other parts before downloading.
Is all the contents of the problem files filled NULL charactor?
Or, did the problem files have only modifying parts?

The first file where I noticed the issue, did completely consist of NUL characters. But most files did only partially contain NUL characters. When I tried to reproduce the issue with 1.80 and 1.79, I did the following:

setup BitTorrent Sync and S3FS on Ubuntu 14.04
setup BitTorrent Sync on Windows machine
put a FLAC file and let it sync
checked with a S3 filebrowser that the file was synced correctly
now altered the FLAC file contents (changed a metadata tag) and waited until synced again
checked the file with a S3 filebrowser again

With 1.80 and 1.79, I did not manage to get a non-corrupt file after sync. With 1.78 I could not reproduce the issue but found some corrupt files after using 1.78 in our test-environment again.

I'm fixing #428(#444) which is a bug related to stats cache file and temporary file.
(If you can, please use the latest codes in master branch.)

#428 (#444) seems to be about a bug when using local caching, right? We don't have local caching enabled so this should not be an issue. I will try out current master anyway and post back.

And I think that we should know about your free disk space.
Does your machine have enough disk free space?
If you do not have enough disk space, you should use ensure_diskfree option.
It will affect the operation of upload and download, and the temporary file.

There are > 3GB of free space left on the machine so this should not be an issue. As mentioned above, local caching is disabled so there are no 'peak times' where disk space gets low. The files being transferred are no bigger than 100MB and when reproducing the issue, I was working with one single file of 18MB in size.

Last, if you can, please run s3fs on forground(-f option), and run with debug option(-dbglevel=XXX).
(If you use the debug option, please note that a large amount of debug messages are output.)
I expect that there is information in the file that failed in the debug message.

I will do and post back.

Thanks for your work!

@ClemensSchneider commented on GitHub (Jul 19, 2016): @ggtakec Thanks for your reply, I will try and answer as detailled as possible: > This issue seems that s3fs failed downloading other parts before downloading. > Is all the contents of the problem files filled NULL charactor? > Or, did the problem files have only modifying parts? The first file where I noticed the issue, did completely consist of NUL characters. But most files did only partially contain NUL characters. When I tried to reproduce the issue with 1.80 and 1.79, I did the following: - setup BitTorrent Sync and S3FS on Ubuntu 14.04 - setup BitTorrent Sync on Windows machine - put a FLAC file and let it sync - checked with a S3 filebrowser that the file was synced correctly - now altered the FLAC file contents (changed a metadata tag) and waited until synced again - checked the file with a S3 filebrowser again With 1.80 and 1.79, I did not manage to get a non-corrupt file after sync. With 1.78 I could not reproduce the issue but found some corrupt files after using 1.78 in our test-environment again. > I'm fixing #428(#444) which is a bug related to stats cache file and temporary file. > (If you can, please use the latest codes in master branch.) #428 (#444) seems to be about a bug when using local caching, right? We don't have local caching enabled so this should not be an issue. I will try out current master anyway and post back. > And I think that we should know about your free disk space. > Does your machine have enough disk free space? > If you do not have enough disk space, you should use ensure_diskfree option. > It will affect the operation of upload and download, and the temporary file. There are > 3GB of free space left on the machine so this should not be an issue. As mentioned above, local caching is disabled so there are no 'peak times' where disk space gets low. The files being transferred are no bigger than 100MB and when reproducing the issue, I was working with one single file of 18MB in size. > Last, if you can, please run s3fs on forground(-f option), and run with debug option(-dbglevel=XXX). > (If you use the debug option, please note that a large amount of debug messages are output.) > I expect that there is information in the file that failed in the debug message. I will do and post back. Thanks for your work!

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jul 20, 2016):

@ggtakec I tested with both master (b76fc35) and 1.80 again. Both leave the files corrupted.
At the time of testing with a single file of 15MB size, 4.5GB of space were left on the disk, so this should not be an issue.

I ran S3FS with debug-level dbg and uploaded both results (for 1.80 and master). I put the original files and the corrupted files along with the logs. You may download the results here:
https://www.dropbox.com/sh/xa6iyvwwhp45rve/AACmApk11ELduhdL36BInvbda?dl=0

The corrupted files are filled with NUL bytes in the end of the track. The modification itself (changed a tag) successfully made it to the file...

Mount options were as follows:
sudo s3fs myBucket /mnt/s3-music-archive -f -odbglevel=dbg,enable_noobj_cache,max_stat_cache_size=10000,stat_cache_expire=300,retries=20,nomultipart,enable_content_md5,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000 >> ~/s3fs_debug.txt 2>&1

@ClemensSchneider commented on GitHub (Jul 20, 2016): @ggtakec I tested with both master (b76fc35) and 1.80 again. Both leave the files corrupted. At the time of testing with a single file of 15MB size, 4.5GB of space were left on the disk, so this should not be an issue. I ran S3FS with debug-level dbg and uploaded both results (for 1.80 and master). I put the original files and the corrupted files along with the logs. You may download the results here: https://www.dropbox.com/sh/xa6iyvwwhp45rve/AACmApk11ELduhdL36BInvbda?dl=0 The corrupted files are filled with NUL bytes in the end of the track. The modification itself (changed a tag) successfully made it to the file... Mount options were as follows: `sudo s3fs myBucket /mnt/s3-music-archive -f -odbglevel=dbg,enable_noobj_cache,max_stat_cache_size=10000,stat_cache_expire=300,retries=20,nomultipart,enable_content_md5,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000 >> ~/s3fs_debug.txt 2>&1`

kerem commented

Author

Owner

@ggtakec commented on GitHub (Jul 21, 2016):

@ClemensSchneider Thanks for your response and results.

But I could not find any error(wrong) in your logs.
And I has confirmed that both corrupted files are filled by null charactor after 32KB, even though those file size are 16MB.

Before I would expect this cause, please let me explain the behavior of s3fs.

s3fs makes a local file before uploading any file(object) to S3.
And s3fs uploads this local file by curl after finishing writing contents data to it.
This logic is the same for using cache files and not using.
(If s3fs does not use the cache, s3fs uses the temporary file instead of the cache)

As you said, it seems that free disk space was not related to this issue.
And the cache files were not too.
This problem is that the file object is uploaded only first 32KB.

I not familiar with BitTorrent Sync, but if it is not synchronized files by each parts(some bytes), this problem would be caused s3fs.

Moreover, if you have any reason please tell me why specify nomultipart option.
I mean that this option significantly changes the behavior of the s3fs.
And if you can, please run s3fs without nomultipart option.

I will examine this cause. please wait a while.

@ggtakec commented on GitHub (Jul 21, 2016): @ClemensSchneider Thanks for your response and results. But I could not find any error(wrong) in your logs. And I has confirmed that both corrupted files are filled by null charactor after 32KB, even though those file size are 16MB. Before I would expect this cause, please let me explain the behavior of s3fs. s3fs makes a local file before uploading any file(object) to S3. And s3fs uploads this local file by curl after finishing writing contents data to it. This logic is the same for using cache files and not using. (If s3fs does not use the cache, s3fs uses the temporary file instead of the cache) As you said, it seems that free disk space was not related to this issue. And the cache files were not too. This problem is that the file object is uploaded only first 32KB. I not familiar with BitTorrent Sync, but if it is not synchronized files by each parts(some bytes), this problem would be caused s3fs. Moreover, if you have any reason please tell me why specify nomultipart option. I mean that this option significantly changes the behavior of the s3fs. And if you can, please run s3fs without nomultipart option. I will examine this cause. please wait a while.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jul 21, 2016):

@ggtakec Thanks for investigating this!

It is possibly related to the way BitTorrent Sync 'patches' files when only parts of the file have been modified. Maybe writing these patches in a high frequency causes this issue to appear?
Please do note, that I can reliably reproduce the issue with 1.79 and 1.80, but not with 1.78.

Moreover, if you have any reason please tell me why specify nomultipart option.
I mean that this option significantly changes the behavior of the s3fs.
And if you can, please run s3fs without nomultipart option.

As noted above, I had S3FS running with multipart-uploads enabled and disabled them in hope to possibly fix the issue. I will retest master with multipart enabled and let you know if it still causes the same issue.

You could maybe better debug the issue by simply installing BitTorrent Sync, select a mounted S3FS folder as target directory, sync one FLAC file (e.g. one of the files in the dropbox-link) and then alter a tag in this FLAC file (using foobar2000 for example).

Let me know if I can be of any futher help.

Thanks!

@ClemensSchneider commented on GitHub (Jul 21, 2016): @ggtakec Thanks for investigating this! It is possibly related to the way BitTorrent Sync 'patches' files when only parts of the file have been modified. Maybe writing these patches in a high frequency causes this issue to appear? Please do note, that I can reliably reproduce the issue with 1.79 and 1.80, but not with 1.78. > Moreover, if you have any reason please tell me why specify nomultipart option. > I mean that this option significantly changes the behavior of the s3fs. > And if you can, please run s3fs without nomultipart option. As noted above, I had S3FS running with multipart-uploads enabled and disabled them in hope to possibly fix the issue. I will retest master with multipart enabled and let you know if it still causes the same issue. You could maybe better debug the issue by simply installing BitTorrent Sync, select a mounted S3FS folder as target directory, sync one FLAC file (e.g. one of the files in the dropbox-link) and then alter a tag in this FLAC file (using foobar2000 for example). Let me know if I can be of any futher help. Thanks!

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jul 21, 2016):

@ggtakec I retried with multipart uploads enabled without any success - result was the same corrupted file.
I performed the same action again with v1.78 and could not reproduce the issue. I uploaded the debug-log to the dropbox-folder mentioned above, too.

Commandline now used was s3fs myBucket /mnt/s3-music-archive -f --debug -oenable_noobj_cache,max_stat_cache_size=10000,stat_cache_expire=300,retries=20,enable_content_md5,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000

There must have been something changed between v1.78 and v1.79 that causes this error.

Thanks!

@ClemensSchneider commented on GitHub (Jul 21, 2016): @ggtakec I retried with multipart uploads enabled without any success - result was the same corrupted file. I performed the same action again with v1.78 and could not reproduce the issue. I uploaded the debug-log to the dropbox-folder mentioned above, too. Commandline now used was `s3fs myBucket /mnt/s3-music-archive -f --debug -oenable_noobj_cache,max_stat_cache_size=10000,stat_cache_expire=300,retries=20,enable_content_md5,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000` There must have been something changed between v1.78 and v1.79 that causes this error. Thanks!

kerem commented

Author

Owner

@ggtakec commented on GitHub (Jul 24, 2016):

@ClemensSchneider Thanks for your report about nomultipart option.
I'm sorry for that I did not reproduce and test this problem.

But I found a log that seems to be associated with this problem after I checked your log.

The log said that s3fs uploaded 16MB file which name is ".sync / xxx ... xxx.SyncPart" and s3fs reuploaded same file after that.
But the second uploading file was written only 32KB.
Probabry, I think that this BitTorrent Sync's operation is correct.
Thus I think that the second uploading file size should be 32KB, but probabry it was uploaded 16MB.

(Why the file stat is not updated, I suspect that s3fs is using the old file size.)
And it is possible that BitTrrent Sync updates its control files(*.SyncPart, etc) from multi machines.

So I think we should try to test unset stat cache using.
But you specified max_stat_cache_size and stat_cache_expire, please try to tren off those.(max_stat_cache_size=0)

Please be careful about it will send many request to S3 for getting object stats.
Thanks in advance for your assistance.

@ggtakec commented on GitHub (Jul 24, 2016): @ClemensSchneider Thanks for your report about nomultipart option. I'm sorry for that I did not reproduce and test this problem. But I found a log that seems to be associated with this problem after I checked your log. The log said that s3fs uploaded 16MB file which name is ".sync / xxx ... xxx.SyncPart" and s3fs reuploaded same file after that. But the second uploading file was written only 32KB. Probabry, I think that this BitTorrent Sync's operation is correct. Thus I think that the second uploading file size should be 32KB, but probabry it was uploaded 16MB. (Why the file stat is not updated, I suspect that s3fs is using the old file size.) And it is possible that BitTrrent Sync updates its control files(*.SyncPart, etc) from multi machines. So I think we should try to test unset stat cache using. But you specified max_stat_cache_size and stat_cache_expire, please try to tren off those.(max_stat_cache_size=0) Please be careful about it will send many request to S3 for getting object stats. Thanks in advance for your assistance.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Jul 28, 2016):

@ggtakec Thanks for your investigations.

I retested with disabled stat_cache but got the same corrupted file. I added the logs (>50MB) to the dropbox-folder.

And it is possible that BitTrrent Sync updates its control files(*.SyncPart, etc) from multi machines.

There is only one peer connected so BitTorrent Sync gets only data from one machine.

Commandline used:

 s3fs myBucketName /mnt/s3-music-archive -f -odbglevel=dbg,enab_noobj_cache,max_stat_cache_size=0,stat_cache_expire=300,retries=20,enable_content_md5,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000 > ~/s3fs_debug.txt 2>&1

Thank you!

@ClemensSchneider commented on GitHub (Jul 28, 2016): @ggtakec Thanks for your investigations. I retested with disabled stat_cache but got the same corrupted file. I added the logs (>50MB) to the dropbox-folder. > And it is possible that BitTrrent Sync updates its control files(*.SyncPart, etc) from multi machines. There is only one peer connected so BitTorrent Sync gets only data from one machine. Commandline used: ``` s3fs myBucketName /mnt/s3-music-archive -f -odbglevel=dbg,enab_noobj_cache,max_stat_cache_size=0,stat_cache_expire=300,retries=20,enable_content_md5,multireq_max=50,allow_other,rw,nonempty,uid=1000,gid=1000 > ~/s3fs_debug.txt 2>&1 ``` Thank you!

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Aug 4, 2016):

@ggtakec Is there anything else I can provide to help diagnose this issue?

@ClemensSchneider commented on GitHub (Aug 4, 2016): @ggtakec Is there anything else I can provide to help diagnose this issue?

kerem commented

Author

Owner

@ggtakec commented on GitHub (Aug 14, 2016):

@ClemensSchneider I'm sorry about I have not been got the reason for this issue yet.

From your logs, it seems that s3fs have succeeded to work the job which is the command from FUSE.
Thus, I did not find anything wrong in those logs about s3fs during writing the files(.flac file(sync target file?), and the bittorrent's management files(.SyncTemp and .SynPart)).
Although I'm not familiar with BitTorrent Sync, it seems that bittorrent works with a management files(.SyncTemp and .SyncPart, etc).

In my previous reply, I said that there is somthing wrong about sending 32KB file size, but it was my fault.
(It was not a target file(.flac) but another management file(.SyncPart).)

I am grateful that you had tested about nomultipart and stat, but we did not get the reson of this problem now.

Then if you can, please try to run bittorrent without s3fs(using local hard disk) and check synced files.
And more, please tell us about yuor s3fs version, and use latest codes in master branch.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Aug 14, 2016): @ClemensSchneider I'm sorry about I have not been got the reason for this issue yet. From your logs, it seems that s3fs have succeeded to work the job which is the command from FUSE. Thus, I did not find anything wrong in those logs about s3fs during writing the files(.flac file(sync target file?), and the bittorrent's management files(.SyncTemp and .SynPart)). Although I'm not familiar with BitTorrent Sync, it seems that bittorrent works with a management files(.SyncTemp and .SyncPart, etc). In my previous reply, I said that there is somthing wrong about sending 32KB file size, but it was my fault. (It was not a target file(.flac) but another management file(.SyncPart).) I am grateful that you had tested about nomultipart and stat, but we did not get the reson of this problem now. Then if you can, please try to run bittorrent without s3fs(using local hard disk) and check synced files. And more, please tell us about yuor s3fs version, and use latest codes in master branch. Thanks in advance for your assistance.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Aug 15, 2016):

@ggtakec Thanks for your efforts.

As I already noted, I could reproduce the issue with the one single test-file with 1.79, 1.80 and latest master. I could not reproduce the issue with 1.78.
So BitTorrent Sync works fine in this case as I didn't change the BitTorrent Sync version.

Maybe I'll have to try and find the commit that introduced this bug using bisect when I'm back from holidays then..

@ClemensSchneider commented on GitHub (Aug 15, 2016): @ggtakec Thanks for your efforts. As I already noted, I could reproduce the issue with the one single test-file with 1.79, 1.80 and latest master. I could not reproduce the issue with 1.78. So BitTorrent Sync works fine in this case as I didn't change the BitTorrent Sync version. Maybe I'll have to try and find the commit that introduced this bug using bisect when I'm back from holidays then..

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Sep 23, 2016):

@ggtakec

I finally found the time to bisect through the commits. The last working commit is d2246297bd .

The commit that follows (114966e7c0) seems to break the sync functionality all together (when trying to sync a file, the file never completely reaches the bucket, only its partial (.bts) counterpart). Then, trying d06b6d7d41, when altering a FLAC file, it is synced to the bucket but its content is corrupt. So either 114966e7c0 or d06b6d7d41 do introduce the bug that is still persistent in current master (tested).

I hope you find a hint there what could cause the file-corruption as it is a major issue for other people trying to use S3FS in a similar way as we do.

Thanks in advance and let me know if you need any further information!

@ClemensSchneider commented on GitHub (Sep 23, 2016): @ggtakec I finally found the time to bisect through the commits. The last working commit is d2246297bd5083810d7838963d1af379cf861048 . The commit that follows (114966e7c0371eed17e2898cdf505ba4dbfb9b59) seems to break the sync functionality all together (when trying to sync a file, the file never completely reaches the bucket, only its partial (.bts) counterpart). Then, trying d06b6d7d419bab2b09b919eb3cbcd63b6f03b941, when altering a FLAC file, it is synced to the bucket but its content is corrupt. So either 114966e7c0371eed17e2898cdf505ba4dbfb9b59 or d06b6d7d419bab2b09b919eb3cbcd63b6f03b941 do introduce the bug that is still persistent in current master (tested). I hope you find a hint there what could cause the file-corruption as it is a major issue for other people trying to use S3FS in a similar way as we do. Thanks in advance and let me know if you need any further information!

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Sep 30, 2016):

@ggtakec

Do you already have an idea which changes could have introduced this bug? Is there anything else I can do from my side?

@ClemensSchneider commented on GitHub (Sep 30, 2016): @ggtakec Do you already have an idea which changes could have introduced this bug? Is there anything else I can do from my side?

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 2, 2016):

@ClemensSchneider Thanks for your response.
I'm going to check the code diff and try to find a bug in that.
Please wait a while.
Thanks in advance for your help.

@ggtakec commented on GitHub (Oct 2, 2016): @ClemensSchneider Thanks for your response. I'm going to check the code diff and try to find a bug in that. Please wait a while. Thanks in advance for your help.

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 10, 2016):

@ClemensSchneider
s3fs had been modified about the local cache at those commits(d224629,114966e, d06b6d7).
Those changes were that s3fs duplicates the opened file handle which is already opened.
I have examined those logic for this issue.

And, I was able to verify the operation(problem) of s3fs by the following procedure.
This behavior is thought to be close to your issue, but I do not have conclusive evidence.

I found a problem with the following procedure:

run s3fs without "use_cache" option.
run two type processes:
A-process) writes 1024 bytes to a file which offset is 0 to 1023.
B-process) writes 1024 bytes which offset is 1024 to 2048.
these processes write new files with following procedure.
3-1) B-process truncates, this initializes new file with 2048 file size.
3-2) A-process writes datas(from 0 to 1023 offset in file)
3-3) B-process writes datas(from 1024 to 2047 offset in file)
3-4) B-process closes the file, it means flushing all date to S3.
3-5) A-process closes the file, it means flushing all date to S3.

After that, the new file on S3 has null data area which offset from 1024 to 2047.
This cause is that s3fs loaded from 1024 to 2047 at (3-2), but the file have not been uploaded on S3 yet.
So that, these area filled null and s3fs over wrote all area at (3-5), thus uploaded broken file.

The bug is that s3fs does not use(duplicate) opened local file handle(temporary file) when use_cache option is not specified.
I uploaded issue#435 branch for this patch.
You can try to test this branch codes.

If you can, please test following two cases in order to confirm that this problem is the cause of this issue.

run s3fs(latest master branch) with "use_cache"
run s3fs which is built from issue#435( https://github.com/s3fs-fuse/s3fs-fuse/tree/issue%23435 ) branch,

I think that if both test are no problem, this problem is the cause of this.
And I check the codes and merge to master branch.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Oct 10, 2016): @ClemensSchneider s3fs had been modified about the local cache at those commits(d224629,114966e, d06b6d7). Those changes were that s3fs duplicates the opened file handle which is already opened. I have examined those logic for this issue. And, I was able to verify the operation(problem) of s3fs by the following procedure. This behavior is thought to be close to your issue, but I do not have conclusive evidence. I found a problem with the following procedure: 1) run s3fs without "use_cache" option. 2) run two type processes: _A-process) writes 1024 bytes to a file which offset is 0 to 1023._ _B-process) writes 1024 bytes which offset is 1024 to 2048._ 3) these processes write new files with following procedure. _3-1) B-process truncates, this initializes new file with 2048 file size._ _3-2) A-process writes datas(from 0 to 1023 offset in file)_ _3-3) B-process writes datas(from 1024 to 2047 offset in file)_ _3-4) B-process closes the file, it means flushing all date to S3._ _3-5) A-process closes the file, it means flushing all date to S3._ After that, the new file on S3 has null data area which offset from 1024 to 2047. This cause is that s3fs loaded from 1024 to 2047 at (3-2), but the file have not been uploaded on S3 yet. So that, these area filled null and s3fs over wrote all area at (3-5), thus uploaded broken file. The bug is that s3fs does not use(duplicate) opened local file handle(temporary file) when use_cache option is not specified. I uploaded issue#435 branch for this patch. You can try to test this branch codes. If you can, please test following two cases in order to confirm that this problem is the cause of this issue. - run s3fs(latest master branch) with "use_cache" - run s3fs which is built from issue#435( https://github.com/s3fs-fuse/s3fs-fuse/tree/issue%23435 ) branch, I think that if both test are no problem, this problem is the cause of this. And I check the codes and merge to master branch. Thanks in advance for your assistance.

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 11, 2016):

I'm sorry about issue#435 branch has a bug, but it was fixed now.
Thanks in advance for your help.

@ggtakec commented on GitHub (Oct 11, 2016): I'm sorry about issue#435 branch has a bug, but it was fixed now. Thanks in advance for your help.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Oct 21, 2016):

@ggtakec
Thanks for investigating!

I built S3FS from issue#435 branch and tried to reproduce the bug:

sync complete FLAC file using BitTorrent Sync
verify that it's valid
change a tag
wait for sync

The file unfortunately still is corrupt. Seems like the NUL content starts earlier in the file now. I've uploaded the corrupted file for reference here: https://www.dropbox.com/s/e1cl3i6mmnnfkkl/P%C3%A1jaro%20Sunrise_Beggar-Lover.flac?dl=0

Thanks for your help and patience!

@ClemensSchneider commented on GitHub (Oct 21, 2016): @ggtakec Thanks for investigating! I built S3FS from issue#435 branch and tried to reproduce the bug: - sync complete FLAC file using BitTorrent Sync - verify that it's valid - change a tag - wait for sync The file unfortunately still is corrupt. Seems like the NUL content starts earlier in the file now. I've uploaded the corrupted file for reference here: https://www.dropbox.com/s/e1cl3i6mmnnfkkl/P%C3%A1jaro%20Sunrise_Beggar-Lover.flac?dl=0 Thanks for your help and patience!

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 23, 2016):

@ClemensSchneider
I will try to be able to reproduce again.
I'm sorry, but please wait a while.
Regards,

@ggtakec commented on GitHub (Oct 23, 2016): @ClemensSchneider I will try to be able to reproduce again. I'm sorry, but please wait a while. Regards,

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 23, 2016):

@ClemensSchneider
The master branch has also been updated a while ago.
Because it has been fixed for Content-Length, if you can please try to test it.
Regards,

@ggtakec commented on GitHub (Oct 23, 2016): @ClemensSchneider The master branch has also been updated a while ago. Because it has been fixed for Content-Length, if you can please try to test it. Regards,

kerem commented

Author

Owner

@ggtakec commented on GitHub (Oct 30, 2016):

@ClemensSchneider
I could not reproduce this issue after I built Bit-Torrent Sync test environment and tested on it.

My environment for this testing is following:

* Ubuntu (Bit-Torrent Sync server)
    Version:    btsync 2.0.0-2

    install:    $ sudo add-apt-repository ppa:tuxpoldo/btsync
                $ sudo apt-get install btsync

    run:        /usr/lib/btsync/btsync-daemon --nodaemon --log sync.log --config /etc/btsync/debconf-default.conf

    Sync dir:   /mnt/btsync
                (this directory is mounted by s3fs)

    s3fs ver:   1.80 and 'issue#435 branch'

    run s3fs:   $ sudo s3fs my-bucket /mnt/btsync -o allow_other,url=https://s3.amazonaws.com,multireq_max=20,dbglevel=info -f
        $ sudo s3fs my-bucket /mnt/btsync -o allow_other,use_cache=/tmp,url=https://s3.amazonaws.com,multireq_max=20,dbglevel=info -f

* Client (Resilio)
    Client:     Windows / Mac(OSX)

    S/W:        https://www.resilio.com/

* Sync
    Windows <---> ubuntu(mount by s3fs) <---> MAC

*Test files
               Over 100 Mbytes.

The directory(/mnt/btsync) which is used by Bit-Torrent Sync is mounted by s3fs.
I used s3fs which version are 1.80 and "issue#435" branch on Github.
And I tried to mount by two option type for s3fs, one of type was using use_cache option, the other is not.

The test files are over 100MBytes, and I compared both sent file and received file after syncing by btsync.
But as a result, there was no difference in both files.

I don't know why this problem does not occur on my environment.
If you found the difference about environment, please let me know.

We will update s3fs to 1.81 soon, but I'm sorry for that we will not be able to fix this issue until launching next version.

Regards.

@ggtakec commented on GitHub (Oct 30, 2016): @ClemensSchneider I could not reproduce this issue after I built Bit-Torrent Sync test environment and tested on it. My environment for this testing is following: ``` * Ubuntu (Bit-Torrent Sync server) Version: btsync 2.0.0-2 install: $ sudo add-apt-repository ppa:tuxpoldo/btsync $ sudo apt-get install btsync run: /usr/lib/btsync/btsync-daemon --nodaemon --log sync.log --config /etc/btsync/debconf-default.conf Sync dir: /mnt/btsync (this directory is mounted by s3fs) s3fs ver: 1.80 and 'issue#435 branch' run s3fs: $ sudo s3fs my-bucket /mnt/btsync -o allow_other,url=https://s3.amazonaws.com,multireq_max=20,dbglevel=info -f $ sudo s3fs my-bucket /mnt/btsync -o allow_other,use_cache=/tmp,url=https://s3.amazonaws.com,multireq_max=20,dbglevel=info -f * Client (Resilio) Client: Windows / Mac(OSX) S/W: https://www.resilio.com/ * Sync Windows <---> ubuntu(mount by s3fs) <---> MAC *Test files Over 100 Mbytes. ``` The directory(/mnt/btsync) which is used by Bit-Torrent Sync is mounted by s3fs. I used s3fs which version are 1.80 and "issue#435" branch on Github. And I tried to mount by two option type for s3fs, one of type was using use_cache option, the other is not. The test files are over 100MBytes, and I compared both sent file and received file after syncing by btsync. But as a result, there was no difference in both files. I don't know why this problem does not occur on my environment. If you found the difference about environment, please let me know. We will update s3fs to 1.81 soon, but I'm sorry for that we will not be able to fix this issue until launching next version. Regards.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Oct 31, 2016):

@ggtakec

Thanks for your efforts. The setup looks good to me except that I use btsync 2.3.7 on both the ubuntu box and the windows client. On the ubuntu box, I installed 2.3.7 using this guide: http://blog.bittorrent.com/2016/02/18/official-linux-packages-for-sync-now-available/
I don't know, if it's still valid after BitTorrent Sync has been renamed to Resilio Sync.

The initial sync has never been the problem. It is only that after changing a small part of a file (e.g. by changing meta-data in a FLAC file) that has already been synced, the issue appears.
Have you tried this scenario?

Download this FLAC file: https://www.dropbox.com/s/gtd2du7ovq0ya8n/ORIGINAL_P%C3%A1jaro%20Sunrise_Beggar-Lover.flac?dl=0
Download a software for editing meta-data, e.g. www.mp3tag.de/en/download.html
Start BTSync
Wait for the FLAC file to be synced and check that it's valid in the bucket
Edit its meta-data by changing one of the tags or add a new tag
Wait for BTSync to have synced
Check the file contents of the FLAC file (now it's corrupted)

Thanks for your help!

@ClemensSchneider commented on GitHub (Oct 31, 2016): @ggtakec Thanks for your efforts. The setup looks good to me except that I use btsync 2.3.7 on both the ubuntu box and the windows client. On the ubuntu box, I installed 2.3.7 using this guide: http://blog.bittorrent.com/2016/02/18/official-linux-packages-for-sync-now-available/ I don't know, if it's still valid after BitTorrent Sync has been renamed to Resilio Sync. The initial sync has never been the problem. It is only that after changing a small part of a file (e.g. by changing meta-data in a FLAC file) that has already been synced, the issue appears. Have you tried this scenario? 1) Download this FLAC file: https://www.dropbox.com/s/gtd2du7ovq0ya8n/ORIGINAL_P%C3%A1jaro%20Sunrise_Beggar-Lover.flac?dl=0 2) Download a software for editing meta-data, e.g. www.mp3tag.de/en/download.html 3) Start BTSync 4) Wait for the FLAC file to be synced and check that it's valid in the bucket 5) Edit its meta-data by changing one of the tags or add a new tag 6) Wait for BTSync to have synced 7) Check the file contents of the FLAC file (now it's corrupted) Thanks for your help!

kerem commented

Author

Owner

@ggtakec commented on GitHub (Nov 6, 2016):

@ClemensSchneider
I was able to reproduce the issue.
This is a bug, I began the investigation of the cause.
Please a little more time to solve this.

Regards,

@ggtakec commented on GitHub (Nov 6, 2016): @ClemensSchneider I was able to reproduce the issue. This is a bug, I began the investigation of the cause. Please a little more time to solve this. Regards,

kerem commented

Author

Owner

@ggtakec commented on GitHub (Nov 19, 2016):

@ClemensSchneider
I merged codes on issue#435 branch, I hope the codes fixed this problem.
Please try to test your bittrrent sync by issue#435 branch codes.

Thanks in advance for your assistance.

@ggtakec commented on GitHub (Nov 19, 2016): @ClemensSchneider I merged codes on issue#435 branch, I hope the codes fixed this problem. Please try to test your bittrrent sync by issue#435 branch codes. Thanks in advance for your assistance.

kerem commented

Author

Owner

@ggtakec commented on GitHub (Dec 4, 2016):

@ClemensSchneider
I merged this fixed branch to master branch now( #511 ), because it seems that this problem is fixed by lastest patch.
I'm going to this issue, and please try to use master branch's codes.

I hope this new master codes works good for you.
Thanks in advance for your assistance.

@ggtakec commented on GitHub (Dec 4, 2016): @ClemensSchneider I merged this fixed branch to master branch now( #511 ), because it seems that this problem is fixed by lastest patch. I'm going to this issue, and please try to use master branch's codes. I hope this new master codes works good for you. Thanks in advance for your assistance.

kerem commented

Author

Owner

@ClemensSchneider commented on GitHub (Dec 4, 2016):

@ggtakec
Sorry for the late reply. I tested master-branch just now and could not reproduce the issue! Awesome, thanks a lot :)

@ClemensSchneider commented on GitHub (Dec 4, 2016): @ggtakec Sorry for the late reply. I tested master-branch just now and could not reproduce the issue! Awesome, thanks a lot :)

kerem commented

Author

Owner

@raj-aws commented on GitHub (Feb 17, 2021):

@ggtakec
We are having similar issue - https://github.com/s3fs-fuse/s3fs-fuse/issues/1574
We are using V1.87 still able to see the issue.
Can you please help

@raj-aws commented on GitHub (Feb 17, 2021): @ggtakec We are having similar issue - https://github.com/s3fs-fuse/s3fs-fuse/issues/1574 We are using V1.87 still able to see the issue. Can you please help

kerem commented