starred/cv4pve-autosnap
1
0
Fork 0
mirror of https://github.com/Corsinvest/cv4pve-autosnap.git synced 2026-04-25 08:55:49 +03:00
Code Issues 2 Projects Releases 10 Packages Wiki Activity

[GH-ISSUE #95] Wait and retry if VM is locked #88

New issue
Closed
opened 2026-02-26 17:44:21 +03:00 by kerem · 5 comments
kerem commented 2026-02-26 17:44:21 +03:00
Owner
Copy link

Originally created by @dani on GitHub (Dec 9, 2024).
Original GitHub issue: https://github.com/Corsinvest/cv4pve-autosnap/issues/95

What happened?

I'm using autosnap on a 10 nodes cluster with ~100 VM running. It mostly works great, but sometimes, it fails when a VM which should be snapshotted is locked. In my case, the lock usually from one of these cases

  • A backup is running
  • Another cv4pve-autosnap is running (eg daily snap vs hourly snap)

This cluster is running on Ceph, which might not be the fastest to create snapshots (although it has more than decent perf globally, 40 NVMe OSD with a 2x25Gbps dedicated network)

I tried to spread the various jobs at different times (eg, hourly runs each hour past 4min, daily at 00:08, weekly at 00:12 on sunday etc. and backups only starts at xxh20), but I still have errors from time to time

Expected behavior

cv4pve-autosnap could wait and retry later if a VM is locked

Relevant log output

No response

Proxmox VE Version

8.2.8

Version (bug)

1.1.11

Version (working)

No response

On what operating system are you experiencing the issue?

Linux

Pull Request

  • I would like to do a Pull Request
Originally created by @dani on GitHub (Dec 9, 2024). Original GitHub issue: https://github.com/Corsinvest/cv4pve-autosnap/issues/95 ### What happened? I'm using autosnap on a 10 nodes cluster with ~100 VM running. It mostly works great, but sometimes, it fails when a VM which should be snapshotted is locked. In my case, the lock usually from one of these cases - A backup is running - Another cv4pve-autosnap is running (eg daily snap vs hourly snap) This cluster is running on Ceph, which might not be the fastest to create snapshots (although it has more than decent perf globally, 40 NVMe OSD with a 2x25Gbps dedicated network) I tried to spread the various jobs at different times (eg, hourly runs each hour past 4min, daily at 00:08, weekly at 00:12 on sunday etc. and backups only starts at xxh20), but I still have errors from time to time ### Expected behavior cv4pve-autosnap could wait and retry later if a VM is locked ### Relevant log output _No response_ ### Proxmox VE Version 8.2.8 ### Version (bug) 1.1.11 ### Version (working) _No response_ ### On what operating system are you experiencing the issue? Linux ### Pull Request - [ ] I would like to do a Pull Request
kerem 2026-02-26 17:44:21 +03:00
kerem commented 2026-02-26 17:44:22 +03:00
Author
Owner
Copy link

@franklupo commented on GitHub (Dec 9, 2024):

If the vm is busy with other operations you can't run snap, even from web GUI. Wait how long?
Why should multiple cv4pve-autosnaps overlap?

<!-- gh-comment-id:2528022970 --> @franklupo commented on GitHub (Dec 9, 2024): If the vm is busy with other operations you can't run snap, even from web GUI. Wait how long? Why should multiple cv4pve-autosnaps overlap?
kerem commented 2026-02-26 17:44:22 +03:00
Author
Owner
Copy link

@dani commented on GitHub (Dec 9, 2024):

I have overlaps because I run several cron, one for each snapshot label, eg

*/30 * * * * root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label frequently --keep 6 --only-running || echo "autosnap failed"
4 * * * * root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label hourly --keep 6 --only-running || echo "autosnap failed"
8 0 * * * root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label daily --keep 4 --only-running || echo "autosnap failed"
12 0 * * 0 root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label weekly --keep 2 --only-running || echo "autosnap failed

On rare occasions, the first job (frequently) is still running when the second (hourly) starts. There's also cases where backups are running (say I'm starting the backup at 20:20, then at 20:30, when the frequently cv4pve-autosnap is triggered, backups are not always done, holding a lock on VM)

<!-- gh-comment-id:2528062349 --> @dani commented on GitHub (Dec 9, 2024): I have overlaps because I run several cron, one for each snapshot label, eg ``` */30 * * * * root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label frequently --keep 6 --only-running || echo "autosnap failed" 4 * * * * root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label hourly --keep 6 --only-running || echo "autosnap failed" 8 0 * * * root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label daily --keep 4 --only-running || echo "autosnap failed" 12 0 * * 0 root systemd-cat /usr/local/bin/cv4pve-autosnap @/etc/pve/priv/autosnap/common.conf snap --label weekly --keep 2 --only-running || echo "autosnap failed ``` On rare occasions, the first job (frequently) is still running when the second (hourly) starts. There's also cases where backups are running (say I'm starting the backup at 20:20, then at 20:30, when the frequently cv4pve-autosnap is triggered, backups are not always done, holding a lock on VM)
kerem commented 2026-02-26 17:44:22 +03:00
Author
Owner
Copy link

@franklupo commented on GitHub (Dec 9, 2024):

yes, even backups fail if a snap is being created. You could use a script that does not take snapshots when there are backups, or take snapshots at the end of the backup.

https://git.proxmox.com/?p=pve-manager.git;a=blob;f=vzdump-hook-script.pl;h=a93eeec80bd09128e70a4a9775438ab658da2191;hb=refs/heads/master

<!-- gh-comment-id:2528071896 --> @franklupo commented on GitHub (Dec 9, 2024): yes, even backups fail if a snap is being created. You could use a script that does not take snapshots when there are backups, or take snapshots at the end of the backup. https://git.proxmox.com/?p=pve-manager.git;a=blob;f=vzdump-hook-script.pl;h=a93eeec80bd09128e70a4a9775438ab658da2191;hb=refs/heads/master
kerem commented 2026-02-26 17:44:22 +03:00
Author
Owner
Copy link

@franklupo commented on GitHub (Jan 3, 2025):

news?

<!-- gh-comment-id:2568923779 --> @franklupo commented on GitHub (Jan 3, 2025): news?
kerem commented 2026-02-26 17:44:22 +03:00
Author
Owner
Copy link

@dani commented on GitHub (Jan 3, 2025):

As a workaround, I now disable autosnap during the backup window

<!-- gh-comment-id:2568943208 --> @dani commented on GitHub (Jan 3, 2025): As a workaround, I now disable autosnap during the backup window
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
improve/v2.0.1
v2.1.1
v2.1.0
v2.0.1
v2.0.0
v1.21.0
v1.20.0
v1.19.0
v1.18.0
v1.17.0
v1.16.0
v1.15.2
v1.15.1
v1.15.0
v1.14.11
v1.14.10
v1.14.9
v1.14.8
v1.14.7
v1.14.6
v1.14.5
v1.14.4
v1.14.3
v1.14.2
v1.14.1
v1.14.0
v1.13.3
v1.13.2
v1.13.1
v1.13.0
v1.12.0
v1.11.0
v1.10.0
v1.9.11
v1.9.10
v1.9.9
v1.9.8
v1.9.7
v1.9.6
v1.9.5
v1.9.4
v1.9.3
v1.9.2
v1.9.1
v1.9.0
v1.8.1
v1.8.0
v1.7.3
v1.7.2
v1.7.1
v1.7.0
v1.6.0
v1.5.0
v1.4.0
v1.3.0
v1.2.0
v1.1.2
1.1.1
0.1.11
0.1.10
0.1.9
0.1.8
0.1.7
0.1.6_all
0.1.5-1_all
0.1.4-1_all
0.1.3-1_all
0.1.2-1_all
0.1.1-1_all
0.1.0-1_all
Labels
Clear labels   
bug

   
enhancement

   
needs investigation

   
needs investigation

   
pull-request

Mirrored from GitHub Pull Request

  
question
No labels
bug
enhancement
needs investigation
needs investigation
pull-request
question
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/cv4pve-autosnap#88
No description provided.