VolumeSnapshotContents are retained after VolumeSnapshots have been deleted #7511
Comments
|
This is expected. VolumeSnapshotContent is first patched to Retain so that after backup namespaced resource VolumeSnapshot is removed so that namespace could be deleted without cascading deletion to (now Retained) VolumeSnapshotContent that is needed to restore from backup.
The only case when volumesnapshotcontent objects will be removed by velero is when backup is expired or deleted.
|
If volumesnapshotcontents are removed as well, velero wouldn't be able to restore your data. Alternatively you can look into https://velero.io/docs/v1.13/csi-snapshot-data-movement/ which removes the need for retained snapshot on cluster by moving data to object store. |
|
I use snapshot data movement, so I expect "VolumeSnapshotContent" objects to always be deleted after the backup and data movement are complete. Here is my Velero schedule: apiVersion: velero.io/v1
kind: Schedule
metadata:
name: nginx-example-backup-every-two-hours
namespace: velero
annotations:
velero.io/csi-volumesnapshot-class_disk.csi.cloud.com: "linstor"
spec:
schedule: "10 0,8-20/2 * * 1-6"
template:
csiSnapshotTimeout: 20m
snapshotVolumes: true
snapshotMoveData: true
includedNamespaces:
- "nginx-example"
includedResources:
- "*"
storageLocation: default
volumeSnapshotLocations:
- default
ttl: 168h0m0s
|
|
hrm.. I'm no expert, but wouldn't you want to ONLY set snapshotMoveData: true and not snapshotVolumes? |
|
Ok based on the bundle file provided at you in fact have not enabled {
"apiVersion": "velero.io/v1",
"kind": "Backup",
"metadata": {
"annotations": {
"velero.io/resource-timeout": "10m0s",
"velero.io/source-cluster-k8s-gitversion": "v1.28.7",
"velero.io/source-cluster-k8s-major-version": "1",
"velero.io/source-cluster-k8s-minor-version": "28"
},
"creationTimestamp": "2024-03-07T18:10:25Z",
"generation": 6,
"labels": {
"kustomize.toolkit.fluxcd.io/name": "stage07",
"kustomize.toolkit.fluxcd.io/namespace": "flux-system",
"velero.io/schedule-name": "velero-backup-every-two-hours",
"velero.io/storage-location": "default"
},
"name": "velero-backup-every-two-hours-20240307181025",
"namespace": "velero",
"resourceVersion": "947441",
"uid": "e714b1df-dea1-445b-bbf5-4d83a50afe01"
},
"spec": {
"csiSnapshotTimeout": "10m0s",
"defaultVolumesToFsBackup": false,
"hooks": {},
"includedNamespaces": [
"velero"
],
"includedResources": [
"*"
],
"itemOperationTimeout": "4h0m0s",
"metadata": {},
"snapshotMoveData": false,
"storageLocation": "default",
"ttl": "168h0m0s",
"volumeSnapshotLocations": [
"default"
]
},
"status": {
"backupItemOperationsAttempted": 2,
"backupItemOperationsCompleted": 2,
"completionTimestamp": "2024-03-07T18:11:26Z",
"csiVolumeSnapshotsAttempted": 1,
"csiVolumeSnapshotsCompleted": 1,
"expiration": "2024-03-14T18:11:14Z",
"formatVersion": "1.1.0",
"hookStatus": {},
"phase": "Completed",
"progress": {
"itemsBackedUp": 152,
"totalItems": 152
},
"startTimestamp": "2024-03-07T18:11:14Z",
"version": 1
}
}, |
|
Schedule for this backup on the bundle file provided which is now paused, still did not have snapshotMoveData set which means the backup generated from schedule will not use snapshotMoveData. {
"apiVersion": "velero.io/v1",
"kind": "Schedule",
"metadata": {
"creationTimestamp": "2024-03-06T10:09:31Z",
"generation": 20,
"labels": {
"kustomize.toolkit.fluxcd.io/name": "stage07",
"kustomize.toolkit.fluxcd.io/namespace": "flux-system"
},
"name": "velero-backup-every-two-hours",
"namespace": "velero",
"resourceVersion": "1453983",
"uid": "36d3badf-b6a4-43ce-b5d6-78a6a4c109cf"
},
"spec": {
"paused": true,
"schedule": "10 0,8-20/2 * * 1-6",
"template": {
"csiSnapshotTimeout": "0s",
"hooks": {},
"includedNamespaces": [
"velero"
],
"includedResources": [
"*"
],
"itemOperationTimeout": "0s",
"metadata": {},
"storageLocation": "default",
"ttl": "168h0m0s",
"volumeSnapshotLocations": [
"default"
]
}
},
"status": {
"lastBackup": "2024-03-08T10:10:26Z",
"phase": "Enabled"
}
} |
|
@kaovilai maybe this issue is due to some weird misconfiguration? I have multiple schedules and only nginx-example should create volume snapshots and move them to S3 storage. However, there is also a velero-backup schedule that should only store one configuration of the velero namespace: apiVersion: velero.io/v1
kind: Schedule
metadata:
name: velero-backup-every-two-hours
namespace: velero
spec:
schedule: '10 0,8-20/2 * * 1-6'
template:
includedNamespaces:
- 'velero'
includedResources:
- '*'
storageLocation: default
volumeSnapshotLocations:
- default
ttl: 168h0m0sBut many of the operations take place in the velero namespace (e.g. data uploads). Now, after you pointed me to the volumesnapshotcontent, which was without volumesnapshot (I actually missed that its name is velero.io/backup-name=velero-backup-every-two-hours-20240307181025 and not nginx-example...) I looked at the backup and saw that the velero-backup contains CSI snapshots of the nginx-example namespace (see below). In the velero-backup-every-two-hours schedule, neither the snapshotMoveData nor snapshotVolumes properties are set. Why does velero-backup get this snapshot from nginx-example? It looks like a temporary snapshot object as part of a nginx-example backup process ! |
I don't think backing up velero namespace was ever recommended. Tho I have not found a dedicated doc saying so. But there have been chats in the past. In the meantime you can add For "syncing schedules" there have been examples of using argocd, we can also reopen #2876 |
Velero does simply what it's being told to do, and it does not have any logic currently that "hey this is velero namespace, treat it differently." |
|
Nothing was actually broken, this is simply code not living up to your expectation but not exactly malfunctioning. |
|
"In the meantime you can add --include-cluster-resources=false to this schedule to avoid said issue." |
|
But to answer the original question, the reason VolumeSnapshotContents are not deleted is that if you're not using datamover, if Velero deletes the VSCs after backup, then it won't be able to restore, since the snapshot bits will be removed. With datamover, VSC contents are copied into the BackupStorageLocation, so VSCs can be deleted post-backup, but without DataMover, the VolumeSnapshotContents are not temporary data, they are required for restore to work. |
@sseago op wants to backup velero namespace. There's only temporary data mover PVC that they don't care about in the namespace. They probably only want velero.io resources. |
|
You probably want a specific short list of included resources, then, excluding everything else. You'd probably want BackupStorageLocations, Secrets, and Schedules. I don't think you'd want Backups/Restores/etc. since those aren't really useful without the related BSL resources, and once you add a BSL, any backups in that BSL are synced to the cluster for you. |
|
I think based on the latest comment by @sseago a solution has been provided. Closing this issue. |
|
@reasonerjt I am personally facing the same issue with the latest version of velero (v1.13.2). Seems that it is happening sporadically with different volumes (sometimes our daily backup goes well, but failed the day after). So I'm not quite sure why this issue has been closed as I don't see any solution in the comments (except excluding the volumes, which obviously is not a good solution) Is it possible that is has something to do with Argo CD autosync? |
What steps did you take and what happened:
I have a Velero schedule that creates a backup of the running application at regular intervals, including the creation of CSI snapshots. The VolumeSnapshotClass used has the deletionPolicy set to Delete. After the backup, I see that there are no VolumeSnapshots, but there are still VolumeSnapshotContents.
What did you expect to happen:
VolumeSnapshotContents are also deleted.
The following information will help us better understand what's going on:
bundle-2024-03-08-13-10-57.tar.gz
Anything else you would like to add:
I see errors in output of the snapshot-controller
Additional command line output
Environment:
velero version):velero client config get features):kubectl version):/etc/os-release): Ubuntu 22.04.4 LTSVote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
The text was updated successfully, but these errors were encountered: