[GH-ISSUE #1259] upload condition #678

New issue

Closed

opened 2026-03-04 01:47:48 +03:00 by kerem · 3 comments

kerem commented

2026-03-04 01:47:48 +03:00

Owner

Originally created by @thierryg on GitHub (Mar 29, 2020).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1259

Version of s3fs being used (s3fs --version)
s3fs --version
Amazon Simple Storage Service File System V1.86 (commit:fe2b269) with OpenSSL
Copyright (C) 2010 Randy Rizun rrizun@gmail.com
License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Kernel information (uname -r)
uname -r
4.4.0-1092-aws

GNU/Linux Distribution, if applicable (cat /etc/os-release)
cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.6 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.6 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

s3fs command line used, if applicable
sudo s3fs dazzl-test-records /data/records -o passwd_file=/etc/passwd-s3fs,sync,endpoint=eu-west-1,allow_other,umask=0007,uid=1001,gid=1001

mount |grep s3fs
s3fs on /data/records type fuse.s3fs (rw,nosuid,nodev,relatime,sync,user_id=0,group_id=0,allow_other)

/etc/fstab entry, if applicable
s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Details about the issue :

I have a process that records video to an s3fs mount point.

The packets are recorded progressively with a fwrite () + fflush ().

When looking at the size of the file in the mount point, with the ls -alh command, I see it increase.
If I look on the amazon web console, the file exists but its size is zero.

Its size changes only when the file is closed.

I do not ask that it be updated in real-time but rather from time to time, according to different thresholds.

Do I believe the files are created in a cache so I wonder on what condition the files are uploaded? Is it only on fclose () or on the size of the files or even that of the cache?

Thanks in advance.
Regards
Thierry

Originally created by @thierryg on GitHub (Mar 29, 2020). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/1259 Version of s3fs being used (s3fs --version) s3fs --version Amazon Simple Storage Service File System V1.86 (commit:fe2b269) with OpenSSL Copyright (C) 2010 Randy Rizun rrizun@gmail.com License GPL2: GNU GPL version 2 https://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Kernel information (uname -r) uname -r 4.4.0-1092-aws GNU/Linux Distribution, if applicable (cat /etc/os-release) cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" cat /etc/os-release NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.6 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial s3fs command line used, if applicable sudo s3fs dazzl-test-records /data/records -o passwd_file=/etc/passwd-s3fs,sync,endpoint=eu-west-1,allow_other,umask=0007,uid=1001,gid=1001 mount |grep s3fs s3fs on /data/records type fuse.s3fs (rw,nosuid,nodev,relatime,sync,user_id=0,group_id=0,allow_other) /etc/fstab entry, if applicable s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) Details about the issue : I have a process that records video to an s3fs mount point. The packets are recorded progressively with a fwrite () + fflush (). When looking at the size of the file in the mount point, with the ls -alh command, I see it increase. If I look on the amazon web console, the file exists but its size is zero. Its size changes only when the file is closed. I do not ask that it be updated in real-time but rather from time to time, according to different thresholds. Do I believe the files are created in a cache so I wonder on what condition the files are uploaded? Is it only on fclose () or on the size of the files or even that of the cache? Thanks in advance. Regards Thierry

kerem

2026-03-04 01:47:48 +03:00

closed this issue
added the
enhancement
label

kerem commented

2026-03-04 01:47:49 +03:00

Author

Owner

@gaul commented on GitHub (Apr 21, 2020):

Currently s3fs uploads a file during either fsync or close. Note that fflush does not call fsync. Agreed that s3fs should have a configurable limit that triggers an fsync.

@gaul commented on GitHub (Apr 21, 2020): Currently s3fs uploads a file during either `fsync` or `close`. Note that `fflush` does not call `fsync`. Agreed that s3fs should have a configurable limit that triggers an `fsync`.

kerem commented

2026-03-04 01:47:49 +03:00

Author

Owner

@gaul commented on GitHub (Oct 10, 2020):

Duplicate of #1257?

@gaul commented on GitHub (Oct 10, 2020): Duplicate of #1257?

kerem commented

2026-03-04 01:47:49 +03:00

Author

Owner

@gaul commented on GitHub (Jan 21, 2021):

s3fs 1.88 added -o max_dirty_data that periodically syncs based on a size threshold. Can you try this and report back?

@gaul commented on GitHub (Jan 21, 2021): s3fs 1.88 added `-o max_dirty_data` that periodically syncs based on a size threshold. Can you try this and report back?

kerem referenced this issue

2026-03-04 02:01:18 +03:00

[PR #688] [MERGED] Improved use of temporary files - #678 #1608