starred/s3fs-fuse
1
0
Fork 0
mirror of https://github.com/s3fs-fuse/s3fs-fuse.git synced 2026-04-25 13:26:00 +03:00
Code Issues 305 Projects Releases 5 Packages Wiki Activity

[GH-ISSUE #113] File cannot upload while writing #71

New issue
Closed
opened 2026-03-04 01:41:46 +03:00 by kerem · 3 comments
kerem commented 2026-03-04 01:41:46 +03:00
Owner
Copy link

Originally created by @wbchn on GitHub (Jan 26, 2015).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/113

I have a problem:

  1. I mount aws s3 as /mnt/s3, and set as a log directory
  2. system write log continuous
    then, i found data write local, but had not upload to s3 until stop write:
ubuntu@ip-home:/mnt/s3/s3fs/my-log$ ll
total 24051
drwxr-xr-x 1 ubuntu ubuntu        0 Jan 25 22:31 ./
drwxr-xr-x 1 ubuntu ubuntu        0 Jan 25 21:35 ../
-rw-r--r-- 1 ubuntu ubuntu 15662121 Jan 26 00:00 my-log-2015-01-25_00000
-rw-r--r-- 1 ubuntu ubuntu  8963246 Jan 26 00:44 my-log-2015-01-26_00000
lrwxrwxrwx 1 ubuntu ubuntu       32 Jan 26 00:44 my-log_current -> 

Found 3 items
-rw-rw-rw-   1   15662121 2015-01-26 00:00 s3://mybucket/s3fs/my-log/my-log-2015-01-25_00000
-rw-rw-rw-   1          0 2015-01-26 00:44 s3://mybucket/s3fs/my-log/my-log-2015-01-26_00000
-rw-rw-rw-   1         32 2015-01-26 00:44 s3://mybucket/s3fs/my-log/my-log_current

When i stop the log writer, all things come ok(looks like without upload from network watch), s3 file same as local.

I tried some method, but none of them works:

  • set without cache (use_cache="")
  • change fd_page_size=10240 (with modify the s3fs.cpp)
  • set max_stat_cache_size=10

Could you help me, how flush or truncate data to s3 while the log is being writen? Thank you very much.

Originally created by @wbchn on GitHub (Jan 26, 2015). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/113 I have a problem: 1. I mount aws s3 as /mnt/s3, and set as a log directory 2. system write log continuous then, **i found data write local, but had not upload to s3 until stop write**: ``` ubuntu@ip-home:/mnt/s3/s3fs/my-log$ ll total 24051 drwxr-xr-x 1 ubuntu ubuntu 0 Jan 25 22:31 ./ drwxr-xr-x 1 ubuntu ubuntu 0 Jan 25 21:35 ../ -rw-r--r-- 1 ubuntu ubuntu 15662121 Jan 26 00:00 my-log-2015-01-25_00000 -rw-r--r-- 1 ubuntu ubuntu 8963246 Jan 26 00:44 my-log-2015-01-26_00000 lrwxrwxrwx 1 ubuntu ubuntu 32 Jan 26 00:44 my-log_current -> Found 3 items -rw-rw-rw- 1 15662121 2015-01-26 00:00 s3://mybucket/s3fs/my-log/my-log-2015-01-25_00000 -rw-rw-rw- 1 0 2015-01-26 00:44 s3://mybucket/s3fs/my-log/my-log-2015-01-26_00000 -rw-rw-rw- 1 32 2015-01-26 00:44 s3://mybucket/s3fs/my-log/my-log_current ``` When i stop the log writer, all things come ok(looks like without upload from network watch), s3 file same as local. I tried some method, but none of them works: - set without cache (use_cache="") - change fd_page_size=10240 (with modify the s3fs.cpp) - set max_stat_cache_size=10 Could you help me, how flush or truncate data to s3 while the log is being writen? Thank you very much.
kerem closed this issue 2026-03-04 01:41:46 +03:00
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@wbchn commented on GitHub (Jan 26, 2015):

This is debug log with -d option:

Jan 25 22:14:46 ip-home kernel: [1109469.594396] fuse init (API version 7.13)
Jan 26 00:44:14 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: last message repeated 4 times
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: last message repeated 6 times
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: last message repeated 12 times
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: last message repeated 4 times
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:15 ip-home s3fs: last message repeated 48 times
Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:15 ip-home s3fs: Body Text: 
Jan 26 00:44:16 ip-home s3fs: last message repeated 47 times
Jan 26 00:44:16 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:16 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:16 ip-home s3fs: Body Text: 
Jan 26 00:44:17 ip-home s3fs: last message repeated 47 times
Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:17 ip-home s3fs: Body Text: 
Jan 26 00:44:17 ip-home s3fs: Body Text: 
Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:17 ip-home s3fs: Body Text: 
Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue.
Jan 26 00:44:17 ip-home s3fs: Body Text: 
Jan 26 00:44:17 ip-home s3fs: last message repeated 2 times
Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue.
<!-- gh-comment-id:71432526 --> @wbchn commented on GitHub (Jan 26, 2015): This is debug log with -d option: ``` Jan 25 22:14:46 ip-home kernel: [1109469.594396] fuse init (API version 7.13) Jan 26 00:44:14 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: last message repeated 4 times Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: last message repeated 6 times Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: last message repeated 12 times Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: last message repeated 4 times Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:15 ip-home s3fs: last message repeated 48 times Jan 26 00:44:15 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:15 ip-home s3fs: Body Text: Jan 26 00:44:16 ip-home s3fs: last message repeated 47 times Jan 26 00:44:16 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:16 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:16 ip-home s3fs: Body Text: Jan 26 00:44:17 ip-home s3fs: last message repeated 47 times Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:17 ip-home s3fs: Body Text: Jan 26 00:44:17 ip-home s3fs: Body Text: Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:17 ip-home s3fs: Body Text: Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue. Jan 26 00:44:17 ip-home s3fs: Body Text: Jan 26 00:44:17 ip-home s3fs: last message repeated 2 times Jan 26 00:44:17 ip-home s3fs: name is file or subdir in dir. but continue. ```
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@ggtakec commented on GitHub (Mar 21, 2015):

Hi
I checked this issue with latest master branch codes, but it works look good.
(The latest codes is merged #138 and #150, those are fixed about cache file discriptor and file syncing.)
I checked that s3fs access any operation when s3fs opens the file under s3fs mount directory and keeps writing to it.
But I could not find any problem about latest codes.
If you can, please try to use latest codes on master branch.

Thanks in advance for your assistant.

<!-- gh-comment-id:84257911 --> @ggtakec commented on GitHub (Mar 21, 2015): Hi I checked this issue with latest master branch codes, but it works look good. (The latest codes is merged #138 and #150, those are fixed about cache file discriptor and file syncing.) I checked that s3fs access any operation when s3fs opens the file under s3fs mount directory and keeps writing to it. But I could not find any problem about latest codes. If you can, please try to use latest codes on master branch. Thanks in advance for your assistant.
kerem commented 2026-03-04 01:41:47 +03:00
Author
Owner
Copy link

@wbchn commented on GitHub (Mar 23, 2015):

Thanks for your reply.

I installed lastest version (I used release v1.7.8 ealier), and it seem working now. The log like this after wrt a record:

s3fs_getattr(722): [...]
    GetStat(171): stat cache hit...
s3fs_flush(2018): ...
    GetStat(171): stat cache hit ....
s3fs_release(2090): ....
    s3fs_release(2115): Warning - file(...),fd(5) is still opened.
<!-- gh-comment-id:84779016 --> @wbchn commented on GitHub (Mar 23, 2015): Thanks for your reply. I installed lastest version (I used release v1.7.8 ealier), and it seem working now. The log like this after wrt a record: ``` s3fs_getattr(722): [...] GetStat(171): stat cache hit... s3fs_flush(2018): ... GetStat(171): stat cache hit .... s3fs_release(2090): .... s3fs_release(2115): Warning - file(...),fd(5) is still opened. ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
v1.97
v1.96
v1.95
v1.94
v1.93
v1.92
v1.91
v1.90
v1.89
v1.88
v1.87
v1.86
v1.85
v1.84
v1.83
v1.82
v1.81
v1.80
v1.79
v1.78
Pre-v1.78
v1.77
v1.76
v1.75
v1.74
Labels
Clear labels   
bug

   
bug

   
dataloss

   
duplicate

   
enhancement

   
feature request

   
help wanted

   
invalid

   
need info

   
performance

   
pull-request

Mirrored from GitHub Pull Request

  
question

   
question

   
testing
No labels
bug
bug
dataloss
duplicate
enhancement
feature request
help wanted
invalid
need info
performance
pull-request
question
question
testing
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/s3fs-fuse#71
No description provided.