starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #428] all my file contents change to ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ #231

New issue
Closed
opened 2026-03-04 01:43:27 +03:00 by kerem · 12 comments
kerem commented 2026-03-04 01:43:27 +03:00
Owner
Copy link

Originally created by @Fei-Guang on GitHub (Jun 6, 2016).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/428

i used 1.79 and started s3fs as
$s3fs mybucket /opt/s3fs -o nomultipart,use_cache=/tmp.
last week , everything seems ok,

the size of under /tmp/ is about 30G
but today ,i removed all cache under /tmp and restart my ubuntu system. then remount
/opt/s3fs, every file contents have changed .

$vi /opt/s3fs/config.txt

^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
~
~
~
~
~
~
~
~
~
"config.txt" [noeol] 1L, 509C

Originally created by @Fei-Guang on GitHub (Jun 6, 2016). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/428 i used 1.79 and started s3fs as $s3fs mybucket /opt/s3fs -o nomultipart,use_cache=/tmp. last week , everything seems ok, the size of under /tmp/ is about 30G but today ,i removed all cache under /tmp and restart my ubuntu system. then remount /opt/s3fs, every file contents have changed . $vi /opt/s3fs/config.txt ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ~ ~ ~ ~ ~ ~ ~ ~ ~ "config.txt" [noeol] 1L, 509C
kerem 2026-03-04 01:43:27 +03:00
  • closed this issue
  • added the
    dataloss
         label
kerem commented 2026-03-04 01:43:28 +03:00
Author
Owner
Copy link

@jamessoubry commented on GitHub (Jun 6, 2016):

We are getting the same issue with 1.79, 1.80 but not 1.78 It seems to only corrupt plane text files when copying to the cache directory

<!-- gh-comment-id:224015637 --> @jamessoubry commented on GitHub (Jun 6, 2016): We are getting the same issue with 1.79, 1.80 but not 1.78 It seems to only corrupt plane text files when copying to the cache directory
kerem commented 2026-03-04 01:43:28 +03:00
Author
Owner
Copy link

@Fei-Guang commented on GitHub (Jun 7, 2016):

do you know why the text file changed to binary file?

<!-- gh-comment-id:224130456 --> @Fei-Guang commented on GitHub (Jun 7, 2016): do you know why the text file changed to binary file?
kerem commented 2026-03-04 01:43:28 +03:00
Author
Owner
Copy link

@jamessoubry commented on GitHub (Jun 8, 2016):

not sure yet. i'm reviewing the diff between 1.78 and 1.79 to try and work out what has broken it but theres a lot of code to go through.

<!-- gh-comment-id:224545472 --> @jamessoubry commented on GitHub (Jun 8, 2016): not sure yet. i'm reviewing the diff between 1.78 and 1.79 to try and work out what has broken it but theres a lot of code to go through.
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 12, 2016):

Hi, @Fei-Guang
The cache file made by s3fs is the contents of the object(file) obtained in blocks.
Range that has not yet been GET does not exist in the cache file and it looks to all non-download area is 0x00.

Maybe, you see those cache file directly, and I think that looking at a range that has not yet been GET.

Regards,

<!-- gh-comment-id:225406300 --> @ggtakec commented on GitHub (Jun 12, 2016): Hi, @Fei-Guang The cache file made by s3fs is the contents of the object(file) obtained in blocks. Range that has not yet been GET does not exist in the cache file and it looks to all non-download area is 0x00. Maybe, you see those cache file directly, and I think that looking at a range that has not yet been GET. Regards,
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@jamessoubry commented on GitHub (Jun 20, 2016):

Hi, @ggtakec @Fei-Guang, I have done extensive testing and have worked out, for us, the issue happens when we delete items from the cache folder without deleting the .stat folder. we originally had a puppet script that deleted items held in the cache for a long time to reduce the cache folder size. It seems this is no longer possible. @Fei-Guang can you test if this is the same issue with you?

<!-- gh-comment-id:227129529 --> @jamessoubry commented on GitHub (Jun 20, 2016): Hi, @ggtakec @Fei-Guang, I have done extensive testing and have worked out, for us, the issue happens when we delete items from the cache folder without deleting the .stat folder. we originally had a puppet script that deleted items held in the cache for a long time to reduce the cache folder size. It seems this is no longer possible. @Fei-Guang can you test if this is the same issue with you?
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jun 29, 2016):

Hi, @jamessoubry
As you said, I was able to reproduce the bug after I deleted the cache file without deleting .stat file.
Since the bug occurs when it is deleted during the upload / download of files, I need the time to fix this.
Please wait a little while.

And thanks for your assistance.

<!-- gh-comment-id:229320729 --> @ggtakec commented on GitHub (Jun 29, 2016): Hi, @jamessoubry As you said, I was able to reproduce the bug after I deleted the cache file without deleting .stat file. Since the bug occurs when it is deleted during the upload / download of files, I need the time to fix this. Please wait a little while. And thanks for your assistance.
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Jul 3, 2016):

@jamessoubry I merged #444, it fixed about removing cache files during upload/download a object.
Please try to use it.
Thanks in advance for your help.

<!-- gh-comment-id:230134335 --> @ggtakec commented on GitHub (Jul 3, 2016): @jamessoubry I merged #444, it fixed about removing cache files during upload/download a object. Please try to use it. Thanks in advance for your help.
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@jamessoubry commented on GitHub (Jul 18, 2016):

@ggtakec Thanks for the help. I will test and get back to you

<!-- gh-comment-id:233284195 --> @jamessoubry commented on GitHub (Jul 18, 2016): @ggtakec Thanks for the help. I will test and get back to you
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@jamessoubry commented on GitHub (Aug 31, 2016):

@ggtakec fyi Ive not got round to testing this yet. will hopefully do it soon

<!-- gh-comment-id:243794535 --> @jamessoubry commented on GitHub (Aug 31, 2016): @ggtakec fyi Ive not got round to testing this yet. will hopefully do it soon
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Dec 4, 2016):

@jamessoubry
I merged codes to master branch for fixing #435 bug which is very similar to this issue.
I fixed this issue with #444, but there may still have been defects.
But now I merged codes by #511, I think that this issue is solved.

I'm going to close this issue, but please reopen or post new issue if the problem continues.

Thanks in advance for your help.

<!-- gh-comment-id:264692488 --> @ggtakec commented on GitHub (Dec 4, 2016): @jamessoubry I merged codes to master branch for fixing #435 bug which is very similar to this issue. I fixed this issue with #444, but there may still have been defects. But now I merged codes by #511, I think that this issue is solved. I'm going to close this issue, but please reopen or post new issue if the problem continues. Thanks in advance for your help.
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@ayush-san commented on GitHub (Dec 5, 2019):

I am using s3fs version 1.85 and facing this issue in only some files where some content is changed to ^@^@^@^@^@^@^@^@^@^@

    def _chain_tasks(self,
                     sqoop_import_task: SqoopImportOperator,
                     process_and_replicate_task: ProcessAndReplicateOperator,
                     table_cleanup_task: TableCleanupOperator,
                     hive_msck_repair_task: Optional[HiveMsckRepairOperator]) -> None:
        sqoop_import_task >> process_and_replicate_task >> table_cleanup_task
        if hive_msck_repair_task:
            proces^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

I am using s3fs entry in fstab to mount my s3 folder

s3fs#BUCKET_NAME:/airflow/ /mnt/airflow fuse _netdev,iam_role=auto,umask=0022,uid=0,gid=0,allow_other

It was working fine but with the new deployment, some files content is getting changed to ^@^@^@^@^@^@^@^@^@^@.

By unmounting and mounting again and restarting the process fixed the issue, but I don't think it's a correct way to proceed

<!-- gh-comment-id:561966471 --> @ayush-san commented on GitHub (Dec 5, 2019): I am using s3fs version 1.85 and facing this issue in only some files where some content is changed to ^@^@^@^@^@^@^@^@^@^@ ``` def _chain_tasks(self, sqoop_import_task: SqoopImportOperator, process_and_replicate_task: ProcessAndReplicateOperator, table_cleanup_task: TableCleanupOperator, hive_msck_repair_task: Optional[HiveMsckRepairOperator]) -> None: sqoop_import_task >> process_and_replicate_task >> table_cleanup_task if hive_msck_repair_task: proces^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ``` I am using s3fs entry in fstab to mount my s3 folder `s3fs#BUCKET_NAME:/airflow/ /mnt/airflow fuse _netdev,iam_role=auto,umask=0022,uid=0,gid=0,allow_other` It was working fine but with the new deployment, some files content is getting changed to ^@^@^@^@^@^@^@^@^@^@. By unmounting and mounting again and restarting the process fixed the issue, but I don't think it's a correct way to proceed
kerem commented 2026-03-04 01:43:29 +03:00
Author
Owner
Copy link

@gaul commented on GitHub (Jan 26, 2020):

Please test with master and open a new issue if your symptoms persist.

<!-- gh-comment-id:578478748 --> @gaul commented on GitHub (Jan 26, 2020): Please test with master and open a new issue if your symptoms persist.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#231
No description provided.