[GH-ISSUE #2043] S3FS: crontab copy issue #1030

New issue

Open

opened 2026-03-04 01:50:48 +03:00 by kerem · 1 comment

kerem commented

2026-03-04 01:50:48 +03:00

Owner

Originally created by @aoleary on GitHub (Oct 3, 2022).
Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2043

Version of s3fs being used (s3fs --version)

Amazon Simple Storage Service File System V1.91

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse)

Name : fuse
Version : 2.9.7
Release : 3.3.1
Architecture: x86_64

Kernel information (uname -r)

5.3.18-150300.59.93-default

GNU/Linux Distribution, if applicable (cat /etc/os-release)

NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"

s3fs command line used, if applicable

s3fs -o rw,multipart_size=50,logfile=/tmp/xxxxxxx/s3fs.log,use_xattr,nonempty,use_cache=/tmp/backup-hana-srv5,allow_other,use_path_request_style,url=https://xxxxxxxxxxxxx/,passwd_file=/home/xxxxxx/bucketKeys/.xxxxxxxxx-passwd-s3fs,dev,suid xxxxxxxx /xxxxxxxxxx/xxxxxxxxxx

/etc/fstab entry, if applicable

s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs)

Nothing in any log file related to the issue..

Details about issue

We save our HANA DB backups to an S3 bucket, which works perfectly well.
We perform copy operations, via crontab, to have older backups available, as a backup security strategy.

1st of each month at 1h30am : We copy our "daily" backup to a "Monthly" backup folder
Every Saturday at 4am : We copy our "daily" backup to a "Weekly" backup folder
Daily at 6am : We move our "daily" backup to a "Daily" backup folder

I find sometimes, but not always that the copy of the weekly/monthly backup is not always performed correctly.

My guess is that there was an issue with the fact the the 1st of the month landed on a saturday, but there is 2.5H each between the "monthly" and "weekly" copy operations, so time should not be an issue. If I copy it manually, it takes no longer than 1H.

Also, we have implemented this on multiple internal DBs in our organisation, and only the DBs with sizes larger than 100GB were affected.

See attached image.
crontab schedule.txt

Any help appreciated.
Kind regards

Originally created by @aoleary on GitHub (Oct 3, 2022). Original GitHub issue: https://github.com/s3fs-fuse/s3fs-fuse/issues/2043 #### Version of s3fs being used (s3fs --version) Amazon Simple Storage Service File System V1.91 #### Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse, dpkg -s fuse) Name : fuse Version : 2.9.7 Release : 3.3.1 Architecture: x86_64 #### Kernel information (uname -r) 5.3.18-150300.59.93-default #### GNU/Linux Distribution, if applicable (cat /etc/os-release) NAME="SLES" VERSION="15-SP3" VERSION_ID="15.3" PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3" #### s3fs command line used, if applicable s3fs -o rw,multipart_size=50,logfile=/tmp/xxxxxxx/s3fs.log,use_xattr,nonempty,use_cache=/tmp/backup-hana-srv5,allow_other,use_path_request_style,url=https://xxxxxxxxxxxxx/,passwd_file=/home/xxxxxx/bucketKeys/.xxxxxxxxx-passwd-s3fs,dev,suid xxxxxxxx /xxxxxxxxxx/xxxxxxxxxx #### /etc/fstab entry, if applicable ``` ``` #### s3fs syslog messages (grep s3fs /var/log/syslog, journalctl | grep s3fs, or s3fs outputs) Nothing in any log file related to the issue.. ### Details about issue We save our HANA DB backups to an S3 bucket, which works perfectly well. We perform copy operations, via crontab, to have older backups available, as a backup security strategy. 1st of each month at 1h30am : We copy our "daily" backup to a "Monthly" backup folder Every Saturday at 4am : We copy our "daily" backup to a "Weekly" backup folder Daily at 6am : We move our "daily" backup to a "Daily" backup folder I find sometimes, but not always that the copy of the weekly/monthly backup is not always performed correctly. My guess is that there was an issue with the fact the the 1st of the month landed on a saturday, but there is 2.5H each between the "monthly" and "weekly" copy operations, so time should not be an issue. If I copy it manually, it takes no longer than 1H. Also, we have implemented this on multiple internal DBs in our organisation, and only the DBs with sizes larger than 100GB were affected. See attached image. [crontab schedule.txt](https://github.com/s3fs-fuse/s3fs-fuse/files/9695404/crontab.schedule.txt) ![Backup copy issues](https://user-images.githubusercontent.com/16956512/193522878-b72a8647-defd-4287-b4cf-141f273c57c9.png) Any help appreciated. Kind regards

kerem commented

2026-03-04 01:50:49 +03:00

Author

Owner

@ggtakec commented on GitHub (Feb 12, 2023):

@aoleary Sorry for my late reply.
s3fs uses local disk for temporary when uploading files.
It will be the directory specified by the use_cache option.

As a possible cause of your problem, is it possible that processing processes are started at the same time during weekly and monthly executions?
If so, is it possible that disk space, CPU, memory, network, etc. are bottlenecks?

The current master branch adds a new mode called streamupload.
I think that option will perform better for large file uploads like yours.
If you can, try it.

@ggtakec commented on GitHub (Feb 12, 2023): @aoleary Sorry for my late reply. s3fs uses local disk for temporary when uploading files. _It will be the directory specified by the use_cache option._ As a possible cause of your problem, is it possible that processing processes are started at the same time during weekly and monthly executions? If so, is it possible that disk space, CPU, memory, network, etc. are bottlenecks? The current master branch adds a new mode called streamupload. I think that option will perform better for large file uploads like yours. If you can, try it.