[GH-ISSUE #1175] 'use_cache' flag corrupt data being read. #616

Closed
opened 2026-03-04 01:47:15 +03:00 by kerem · 6 comments
Owner

Originally created by @Teej42 on GitHub (Oct 16, 2019).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1175

Version of s3fs being used (s3fs --version)

V1.85

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2

Kernel information (uname -r)

4.15.0-65-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

CentOS Linux 7 (Core)

s3fs command line used, if applicable

Working:

s3fs [bucket] /app/s3fs/mount/[bucket] -o passwd_file=[passwd_file] -o sigv2 -o use_path_request_style -o url=https://s3.us-south.cloud-object-storage.appdomain.cloud -o kernel_cache -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=30 -o multireq_max=30 -o max_background=1000 -o dbglevel=warn -o allow_other

Broken:

s3fs [bucket] /app/s3fs/mount/[bucket] -o passwd_file=[passwd_file] -o use_cache=/mnt/scratchspace/storage/ -o sigv2 -o use_path_request_style -o url=https://s3.us-south.cloud-object-storage.appdomain.cloud -o kernel_cache -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=30 -o multireq_max=30 -o max_background=1000 -o dbglevel=warn -o allow_other

Details about issue

I am using S3FS with IBM Information Server's PX Engine to communicate with IBM COS buckets. The working commands function as expected, the data are accurate, but it eats up the root directory, which is a bad thing in a Kubernetes environment (running out of / space? K8 nuke the pod.)

So I decided to try to use use_cache to redirect the cache to a mounted PVC. But each and every time I do so, I would get the following PX Engine error message in a random manner:

##E IIS-DSEE-TOIX-00159 20:09:15(000) <import(0),21> Short read encountered on import; this most likely indicates one of the following possibilities:
1) the import schema you specified is incorrect
2) invalid data (the schema is correct, but there is an   error in the data).
##E IIS-DSEE-TFRS-00043 20:09:15(001) <import(0),21> Expected 590 bytes, got 160.
##E IIS-DSEE-TOIX-00179 20:09:15(002) <import(0),21> Import error at record 171278.

To translate the error message, the first operator called import is reading text data on partition node 22 (starts from 0), expecting a fixed length record of 590 bytes, and was only given 160 bytes. Note, this behavior is visibly random - this error messages happens at a random number of record, random number of partition node, and a random length. Perhaps with some careful review, we could nail down a pattern out of this.

However, I can attest that running the working mount command would never trigger this issue. Adding use_cache would trigger this error each and every time I attempt to run the same job. Reverting back to the working mount command restores the working condition.

I tried the del_cache option, and it did not change anything. I have not tried any other options yet as I can not see anything relevant to this issue.

The configuration settings are based on the review of this page: https://medium.com/@ozeri/file-like-access-to-ibm-cloud-object-storage-using-s3fs-f5094ed42594

Please advise.

Originally created by @Teej42 on GitHub (Oct 16, 2019). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1175 #### Version of s3fs being used (s3fs --version) V1.85 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) 2.9.2 #### Kernel information (uname -r) 4.15.0-65-generic #### GNU/Linux Distribution, if applicable (cat /etc/os-release) CentOS Linux 7 (Core) #### s3fs command line used, if applicable Working: ``` s3fs [bucket] /app/s3fs/mount/[bucket] -o passwd_file=[passwd_file] -o sigv2 -o use_path_request_style -o url=https://s3.us-south.cloud-object-storage.appdomain.cloud -o kernel_cache -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=30 -o multireq_max=30 -o max_background=1000 -o dbglevel=warn -o allow_other ``` Broken: ``` s3fs [bucket] /app/s3fs/mount/[bucket] -o passwd_file=[passwd_file] -o use_cache=/mnt/scratchspace/storage/ -o sigv2 -o use_path_request_style -o url=https://s3.us-south.cloud-object-storage.appdomain.cloud -o kernel_cache -o max_stat_cache_size=100000 -o multipart_size=52 -o parallel_count=30 -o multireq_max=30 -o max_background=1000 -o dbglevel=warn -o allow_other ``` ### Details about issue I am using S3FS with IBM Information Server's PX Engine to communicate with IBM COS buckets. The working commands function as expected, the data are accurate, but it eats up the root directory, which is a bad thing in a Kubernetes environment (running out of `/` space? K8 nuke the pod.) So I decided to try to use `use_cache` to redirect the cache to a mounted PVC. But each and every time I do so, I would get the following PX Engine error message in a random manner: ``` ##E IIS-DSEE-TOIX-00159 20:09:15(000) <import(0),21> Short read encountered on import; this most likely indicates one of the following possibilities: 1) the import schema you specified is incorrect 2) invalid data (the schema is correct, but there is an error in the data). ##E IIS-DSEE-TFRS-00043 20:09:15(001) <import(0),21> Expected 590 bytes, got 160. ##E IIS-DSEE-TOIX-00179 20:09:15(002) <import(0),21> Import error at record 171278. ``` To translate the error message, the first operator called `import` is reading text data on partition node 22 (starts from 0), expecting a fixed length record of 590 bytes, and was only given 160 bytes. Note, this behavior is visibly random - this error messages happens at a random number of record, random number of partition node, and a random length. Perhaps with some careful review, we could nail down a pattern out of this. However, I can attest that running the working mount command would never trigger this issue. Adding `use_cache` would trigger this error each and every time I attempt to run the same job. Reverting back to the working mount command restores the working condition. I tried the `del_cache` option, and it did not change anything. I have not tried any other options yet as I can not see anything relevant to this issue. The configuration settings are based on the review of this page: https://medium.com/@ozeri/file-like-access-to-ibm-cloud-object-storage-using-s3fs-f5094ed42594 Please advise.
kerem 2026-03-04 01:47:15 +03:00
  • closed this issue
  • added the
    dataloss
    label
Author
Owner

@senfbrot commented on GitHub (Oct 24, 2019):

We are facing a similar problem by using Centos as well.

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.85(commit:unknown) with OpenSSL

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Version     : 2.9.2
Release     : 11.el7
Architecture: x86_64

Kernel information (uname -r)

3.10.0-1062.4.1.el7.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

CentOS Linux 7 (Core)

/etc/fstab entry, if applicable

Working:

s3fs#[bucket1]:/[prefixA]  /mnt/prefixA/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,uid=uidx1234,gid=sshusers,umask=002 0 0
s3fs#[bucket2]:/[prefixB]  /mnt/prefixB/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,uid=uidx1234,gid=sshusers,umask=002 0 0

Not working:

s3fs#[bucket1]:/[prefixA]  /mnt/prefixA/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,use_cache=/tmp/s3fs-cache/,ensure_diskfree=5000,uid=uidx1234,gid=sshusers,umask=002 0 0
s3fs#[bucket2]:/[prefixB]  /mnt/prefixB/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,use_cache=/tmp/s3fs-cache/,ensure_diskfree=5000,uid=uidx1234,gid=sshusers,umask=002 0 0

Details about issue

When I copy a file from one bucket to another bucket through s3fs, then the resulting file is in most cases corrupt. The file size of source and target file is identical, but the checksums differ.

cp /mnt/prefixA/file_1.5gb /mnt/prefixB/
md5sum /mnt/prefixA/file_1.5gb
md5sum /mnt/prefixB/file_1.5gb

When use_cache is disabled, then there is no problem.

I've tried the same on an Amazon Linux instance, there seems to be no issue with using the cache or not.

<!-- gh-comment-id:545897600 --> @senfbrot commented on GitHub (Oct 24, 2019): We are facing a similar problem by using Centos as well. #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.85(commit:unknown) with OpenSSL #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) ``` Version : 2.9.2 Release : 11.el7 Architecture: x86_64 ``` #### Kernel information (uname -r) 3.10.0-1062.4.1.el7.x86_64 #### GNU/Linux Distribution, if applicable (cat /etc/os-release) CentOS Linux 7 (Core) #### /etc/fstab entry, if applicable Working: ``` s3fs#[bucket1]:/[prefixA] /mnt/prefixA/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,uid=uidx1234,gid=sshusers,umask=002 0 0 s3fs#[bucket2]:/[prefixB] /mnt/prefixB/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,uid=uidx1234,gid=sshusers,umask=002 0 0 ``` **Not** working: ``` s3fs#[bucket1]:/[prefixA] /mnt/prefixA/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,use_cache=/tmp/s3fs-cache/,ensure_diskfree=5000,uid=uidx1234,gid=sshusers,umask=002 0 0 s3fs#[bucket2]:/[prefixB] /mnt/prefixB/ fuse rw,_netdev,allow_other,endpoint=eu-central-1,iam_role=auto,use_cache=/tmp/s3fs-cache/,ensure_diskfree=5000,uid=uidx1234,gid=sshusers,umask=002 0 0 ``` ### Details about issue When I copy a file from one bucket to another bucket through s3fs, then the resulting file is in most cases corrupt. The file size of source and target file is identical, but the checksums differ. ``` cp /mnt/prefixA/file_1.5gb /mnt/prefixB/ md5sum /mnt/prefixA/file_1.5gb md5sum /mnt/prefixB/file_1.5gb ``` When `use_cache` is disabled, then there is no problem. I've tried the same on an Amazon Linux instance, there seems to be no issue with using the cache or not.
Author
Owner

@gaul commented on GitHub (Feb 3, 2020):

Could you test with master? It has some data corruption fixes. Otherwise we need some kind of way to reproduce the error. The copy command from https://github.com/s3fs-fuse/s3fs-fuse/issues/1175#issuecomment-545897600 seems promising; does this always happen?

<!-- gh-comment-id:581229703 --> @gaul commented on GitHub (Feb 3, 2020): Could you test with master? It has some data corruption fixes. Otherwise we need some kind of way to reproduce the error. The copy command from https://github.com/s3fs-fuse/s3fs-fuse/issues/1175#issuecomment-545897600 seems promising; does this always happen?
Author
Owner

@Teej42 commented on GitHub (Feb 4, 2020):

What should be the build instruction for MacOS, so I can build from master to test this (and the other story I opened)? I would prefers ensuring this build stay separate from the home-brew version I have already installed:

$ s3fs --version
Amazon Simple Storage Service File System V1.85(commit:unknown) with GnuTLS(gcrypt)
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please let me know. Thanks!

<!-- gh-comment-id:581875124 --> @Teej42 commented on GitHub (Feb 4, 2020): What should be the build instruction for MacOS, so I can build from master to test this (and the other story I opened)? I would prefers ensuring this build stay separate from the home-brew version I have already installed: ``` $ s3fs --version Amazon Simple Storage Service File System V1.85(commit:unknown) with GnuTLS(gcrypt) Copyright (C) 2010 Randy Rizun <rrizun@gmail.com> License GPL2: GNU GPL version 2 <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. ``` Please let me know. Thanks!
Author
Owner

@gaul commented on GitHub (Feb 4, 2020):

You can follow these instructions:

https://github.com/s3fs-fuse/s3fs-fuse/blob/master/COMPILATION.md

Don't run make install at the end and you can run the binary directly from src/s3fs.

<!-- gh-comment-id:581899436 --> @gaul commented on GitHub (Feb 4, 2020): You can follow these instructions: https://github.com/s3fs-fuse/s3fs-fuse/blob/master/COMPILATION.md Don't run `make install` at the end and you can run the binary directly from `src/s3fs`.
Author
Owner

@Teej42 commented on GitHub (Feb 5, 2020):

Unfortunately, the instruction is not complete for the MacOS platform -

During Configure, I got the message, No package 'libcrypto' found - and only this page would provide the solution that worked for me: https://github.com/scipr-lab/libsnark/issues/99

LD_LIBRARY_PATH=/usr/local/opt/openssl/lib:"${LD_LIBRARY_PATH}"                    
CPATH=/usr/local/opt/openssl/include:"${CPATH}"                                    
PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig:"${PKG_CONFIG_PATH}"          
export LD_LIBRARY_PATH CPATH PKG_CONFIG_PATH       

Then when I try to make - I get

s3fs.cpp:30:10: fatal error: 'libxml/xpath.h' file not found
#include <libxml/xpath.h>
         ^~~~~~~~~~~~~~~~
1 error generated.

Unfortunately, there is a number of libxml/xpath.h installed all over my system:

$ find / -name xpath.h 2>/dev/null
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2/libxml/xpath.h
/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Signal.app/Contents/Resources/app.asar.unpacked/node_modules/sharp/vendor/include/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/libxml2/libxml/xpath.h
/System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Signal.app/Contents/Resources/app.asar.unpacked/node_modules/sharp/vendor/include/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/libxml2/libxml/xpath.h
/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/libxml2/libxml/xpath.h

I saw this: https://github.com/s3fs-fuse/s3fs-fuse/issues/296

But I get this:

$ xcode-select --install
xcode-select: error: command line tools are already installed, use "Software Update" to install updates

I am not much of a MacOS developer, so any advice would be welcome.

<!-- gh-comment-id:582548983 --> @Teej42 commented on GitHub (Feb 5, 2020): Unfortunately, the instruction is not complete for the MacOS platform - During Configure, I got the message, `No package 'libcrypto' found` - and only this page would provide the solution that worked for me: https://github.com/scipr-lab/libsnark/issues/99 ``` LD_LIBRARY_PATH=/usr/local/opt/openssl/lib:"${LD_LIBRARY_PATH}" CPATH=/usr/local/opt/openssl/include:"${CPATH}" PKG_CONFIG_PATH=/usr/local/opt/openssl/lib/pkgconfig:"${PKG_CONFIG_PATH}" export LD_LIBRARY_PATH CPATH PKG_CONFIG_PATH ``` Then when I try to `make` - I get ``` s3fs.cpp:30:10: fatal error: 'libxml/xpath.h' file not found #include <libxml/xpath.h> ^~~~~~~~~~~~~~~~ 1 error generated. ``` Unfortunately, there is a number of libxml/xpath.h installed all over my system: ``` $ find / -name xpath.h 2>/dev/null /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2/libxml/xpath.h /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Signal.app/Contents/Resources/app.asar.unpacked/node_modules/sharp/vendor/include/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/libxml2/libxml/xpath.h /System/Volumes/Data/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Signal.app/Contents/Resources/app.asar.unpacked/node_modules/sharp/vendor/include/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVOS.platform/Developer/SDKs/AppleTVOS.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/WatchOS.platform/Developer/SDKs/WatchOS.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/WatchSimulator.platform/Developer/SDKs/WatchSimulator.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/AppleTVSimulator.platform/Developer/SDKs/AppleTVSimulator.sdk/usr/include/libxml2/libxml/xpath.h /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator.sdk/usr/include/libxml2/libxml/xpath.h ``` I saw this: https://github.com/s3fs-fuse/s3fs-fuse/issues/296 But I get this: ``` $ xcode-select --install xcode-select: error: command line tools are already installed, use "Software Update" to install updates ``` I am not much of a MacOS developer, so any advice would be welcome.
Author
Owner

@gaul commented on GitHub (Jun 4, 2020):

It might be easier to use the compilation tools from Homebrew. We added some additional inode sanity checks; please test again with master and reopen if symptoms persist.

<!-- gh-comment-id:638856474 --> @gaul commented on GitHub (Jun 4, 2020): It might be easier to use the compilation tools from Homebrew. We added some additional inode sanity checks; please test again with master and reopen if symptoms persist.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#616
No description provided.