starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 21:35:58 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #1811] Failed to truncate open and modified files to zero length #925

New issue
Open
opened 2026-03-04 01:49:59 +03:00 by kerem · 0 comments
kerem commented 2026-03-04 01:49:59 +03:00
Owner
Copy link

Originally created by @VVoidV on GitHub (Dec 1, 2021).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1811

Version of s3fs being used (s3fs --version)

1.90

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

2.9.2

Kernel information (uname -r)

5.4.140-1.el7.elrepo.x86_64

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

s3fs command line used, if applicable

/usr/bin/s3fs -o dbglevel=err,use_path_request_style,allow_other,del_cache,connect_timeout=300,readwrite_timeout=120,url=http://xxx/,use_cache=/s3fs_cache/136088a6-6ae7-4560-a1ce-fe8851fabddf qwer /pref1 /exports/qwer_pref1_0zo9

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

   7703 2021-11-30T08:54:39.699Z [INF] s3fs.cpp:s3fs_truncate(2137): [path=/small8/vdb_control.file][size=0]
   7704 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_parent_object_access(641): [path=/small8/vdb_control.file]
   7705 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_object_access(535): [path=/small8]
   7706 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/small8]
   7707 2021-11-30T08:54:39.699Z [DBG] cache.cpp:GetStat(293): stat cache hit [path=/small8/][time=89793.286427219][hit count=53]
   7708 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_object_access(535): [path=/]
   7709 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/]
   7710 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_object_access(535): [path=/small8/vdb_control.file]
   7711 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/small8/vdb_control.file]
   7712 2021-11-30T08:54:39.699Z [DBG] cache.cpp:GetStat(293): stat cache hit [path=/small8/vdb_control.file][time=89795.528427219][hit count=2]
   7713 2021-11-30T08:54:39.700Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/small8/vdb_control.file]
   7714 2021-11-30T08:54:39.700Z [DBG] cache.cpp:GetStat(293): stat cache hit [path=/small8/vdb_control.file][time=89795.528427219][hit count=3]
   7715 2021-11-30T08:54:39.700Z [DBG] fdcache.cpp:Open(446): [path=/small8/vdb_control.file][size=0][time=-1]
   7716 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Dup(246): [path=/small8/vdb_control.file][fd=9][refcnt=2]
   7717 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Open(315): [path=/small8/vdb_control.file][fd=9][size=-1][time=-1]
   7718 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Dup(246): [path=/small8/vdb_control.file][fd=9][refcnt=3]
   7719 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Close(200): [path=/small8/vdb_control.file][fd=9][refcnt=2]
   7720 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Load(902): [path=/small8/vdb_control.file][fd=9][offset=0][size=0]
   7721 2021-11-30T08:54:39.700Z [INF]       fdcache_entity.cpp:RowFlush(1194): [tpath=][path=/small8/vdb_control.file][fd=9]
   7722 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Load(902): [path=/small8/vdb_control.file][fd=9][offset=0][size=0]

Details about issue

#1631 allow truncating open and modified file to larger size, but truncating to smaller size failed silently.

7703 2021-11-30T08:54:39.699Z [INF] s3fs.cpp:s3fs_truncate(2137): [path=/small8/vdb_control.file][size=0]
....
7715 2021-11-30T08:54:39.700Z [DBG] fdcache.cpp:Open(446): [path=/small8/vdb_control.file][size=0][time=-1]
7716 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Dup(246): [path=/small8/vdb_control.file][fd=9][refcnt=2]
7717 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Open(315): [path=/small8/vdb_control.file][fd=9][size=-1][time=-1]

the code related to this issue

github.com/s3fs-fuse/s3fs-fuse@85ca2a3e45/src/fdcache.cpp (L554-L560)

How about adding a parameter to AutoFdEntity::Open() and FdManager::Open() to tell we are truncating the file to bypass this check?

Example test case

test code

import os

path = "/exports/qwer_pref1_0zo9/"
#path = "/root/"
filename = path + "test_file"
fd = os.open(filename, os.O_CREAT|os.O_RDWR)
try:
    os.write(fd, 'a' * 42)
    print(os.fstat(fd))
    os.ftruncate(fd, 0)
    print(os.fstat(fd))
    os.lseek(fd, 0, os.SEEK_SET)
    os.write(fd, 'b'*10)
finally:
    os.close(fd)

print(os.stat(filename).st_size)

output with s3fs

[root@localhost s3fs_test]# python test_truncate.py
posix.stat_result(st_mode=33261, st_ino=2, st_dev=46L, st_nlink=1, st_uid=0, st_gid=0, st_size=42, st_atime=1638342636, st_mtime=1638342931, st_ctime=1638342931)
posix.stat_result(st_mode=33261, st_ino=2, st_dev=46L, st_nlink=1, st_uid=0, st_gid=0, st_size=42, st_atime=1638342636, st_mtime=1638342931, st_ctime=1638342931)
42
[root@localhost s3fs_test]# cat /exports/qwer_pref1_0zo9/test_file
bbbbbbbbbbaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa[root@localhost s3fs_test]#

output with xfs

[root@localhost s3fs_test]# python test_truncate.py
posix.stat_result(st_mode=33261, st_ino=67164040, st_dev=64771L, st_nlink=1, st_uid=0, st_gid=0, st_size=42, st_atime=1638343274, st_mtime=1638343278, st_ctime=1638343278)
posix.stat_result(st_mode=33261, st_ino=67164040, st_dev=64771L, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1638343274, st_mtime=1638343278, st_ctime=1638343278)
10
[root@localhost s3fs_test]# cat ~/test_file
bbbbbbbbbb[root@localhost s3fs_test]#
Originally created by @VVoidV on GitHub (Dec 1, 2021). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1811 #### Version of s3fs being used (s3fs --version) _1.90_ #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) _2.9.2_ #### Kernel information (uname -r) _5.4.140-1.el7.elrepo.x86_64_ #### GNU/Linux Distribution, if applicable (cat /etc/os-release) ``` NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/" CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7" ``` #### s3fs command line used, if applicable ``` /usr/bin/s3fs -o dbglevel=err,use_path_request_style,allow_other,del_cache,connect_timeout=300,readwrite_timeout=120,url=http://xxx/,use_cache=/s3fs_cache/136088a6-6ae7-4560-a1ce-fe8851fabddf qwer /pref1 /exports/qwer_pref1_0zo9 ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) ``` 7703 2021-11-30T08:54:39.699Z [INF] s3fs.cpp:s3fs_truncate(2137): [path=/small8/vdb_control.file][size=0] 7704 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_parent_object_access(641): [path=/small8/vdb_control.file] 7705 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_object_access(535): [path=/small8] 7706 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/small8] 7707 2021-11-30T08:54:39.699Z [DBG] cache.cpp:GetStat(293): stat cache hit [path=/small8/][time=89793.286427219][hit count=53] 7708 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_object_access(535): [path=/] 7709 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/] 7710 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:check_object_access(535): [path=/small8/vdb_control.file] 7711 2021-11-30T08:54:39.699Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/small8/vdb_control.file] 7712 2021-11-30T08:54:39.699Z [DBG] cache.cpp:GetStat(293): stat cache hit [path=/small8/vdb_control.file][time=89795.528427219][hit count=2] 7713 2021-11-30T08:54:39.700Z [DBG] s3fs.cpp:get_object_attribute(365): [path=/small8/vdb_control.file] 7714 2021-11-30T08:54:39.700Z [DBG] cache.cpp:GetStat(293): stat cache hit [path=/small8/vdb_control.file][time=89795.528427219][hit count=3] 7715 2021-11-30T08:54:39.700Z [DBG] fdcache.cpp:Open(446): [path=/small8/vdb_control.file][size=0][time=-1] 7716 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Dup(246): [path=/small8/vdb_control.file][fd=9][refcnt=2] 7717 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Open(315): [path=/small8/vdb_control.file][fd=9][size=-1][time=-1] 7718 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Dup(246): [path=/small8/vdb_control.file][fd=9][refcnt=3] 7719 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Close(200): [path=/small8/vdb_control.file][fd=9][refcnt=2] 7720 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Load(902): [path=/small8/vdb_control.file][fd=9][offset=0][size=0] 7721 2021-11-30T08:54:39.700Z [INF] fdcache_entity.cpp:RowFlush(1194): [tpath=][path=/small8/vdb_control.file][fd=9] 7722 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Load(902): [path=/small8/vdb_control.file][fd=9][offset=0][size=0] ``` ### Details about issue #1631 allow truncating open and modified file to larger size, but truncating to smaller size failed silently. **7703 2021-11-30T08:54:39.699Z [INF] s3fs.cpp:s3fs_truncate(2137): [path=/small8/vdb_control.file][size=0]** .... **7715 2021-11-30T08:54:39.700Z [DBG] fdcache.cpp:Open(446): [path=/small8/vdb_control.file][size=0][time=-1]** 7716 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Dup(246): [path=/small8/vdb_control.file][fd=9][refcnt=2] **7717 2021-11-30T08:54:39.700Z [DBG] fdcache_entity.cpp:Open(315): [path=/small8/vdb_control.file][fd=9][size=-1][time=-1]** #### the code related to this issue https://github.com/s3fs-fuse/s3fs-fuse/blob/85ca2a3e455c6bc57ee1132b7f3bb4011fbb77d0/src/fdcache.cpp#L554-L560 How about adding a parameter to AutoFdEntity::Open() and FdManager::Open() to tell we are truncating the file to bypass this check? ## Example test case ### test code ``` python import os path = "/exports/qwer_pref1_0zo9/" #path = "/root/" filename = path + "test_file" fd = os.open(filename, os.O_CREAT|os.O_RDWR) try: os.write(fd, 'a' * 42) print(os.fstat(fd)) os.ftruncate(fd, 0) print(os.fstat(fd)) os.lseek(fd, 0, os.SEEK_SET) os.write(fd, 'b'*10) finally: os.close(fd) print(os.stat(filename).st_size) ``` ### output with s3fs ``` [root@localhost s3fs_test]# python test_truncate.py posix.stat_result(st_mode=33261, st_ino=2, st_dev=46L, st_nlink=1, st_uid=0, st_gid=0, st_size=42, st_atime=1638342636, st_mtime=1638342931, st_ctime=1638342931) posix.stat_result(st_mode=33261, st_ino=2, st_dev=46L, st_nlink=1, st_uid=0, st_gid=0, st_size=42, st_atime=1638342636, st_mtime=1638342931, st_ctime=1638342931) 42 [root@localhost s3fs_test]# cat /exports/qwer_pref1_0zo9/test_file bbbbbbbbbbaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa[root@localhost s3fs_test]# ``` ### output with xfs ``` [root@localhost s3fs_test]# python test_truncate.py posix.stat_result(st_mode=33261, st_ino=67164040, st_dev=64771L, st_nlink=1, st_uid=0, st_gid=0, st_size=42, st_atime=1638343274, st_mtime=1638343278, st_ctime=1638343278) posix.stat_result(st_mode=33261, st_ino=67164040, st_dev=64771L, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1638343274, st_mtime=1638343278, st_ctime=1638343278) 10 [root@localhost s3fs_test]# cat ~/test_file bbbbbbbbbb[root@localhost s3fs_test]# ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#925
No description provided.