Error while backup due to OpenEBS lvm-localpv not setting VolumeSnapshot RestoreSize properly

[saggiyogesh]

Hello, getting this error in controller pod. Can you help in identifying the cause.

{"time":"2024-09-15T15:03:48.339839244Z","level":"ERROR","msg":"Reconciler error","controller":"pvcbackup","controllerGroup":"backsnap.skyb.it","controllerKind":"PVCBackup","PVCBackup":{"name":"csi-lvmpv-1726406899","namespace":"default"},"namespace":"default","name":"csi-lvmpv-1726406899","reconcileID":"20828353-229d-426a-a9f0-e2ed970dc1a1","err":"PersistentVolumeClaim \"csi-lvmpv-1726406899\" is invalid: spec.resources[storage]: Invalid value: \"0\": must be greater than zero"}

[sgielen]

Is it possible that you have your PVCRestore spec.targetPvcSize unset or set to zero?

Hold on, that's if you're restoring, but it looks like this is still during a backup. Let me dig a little deeper

[sgielen]

So the backup process is roughly as follows:

1. Retrieve information about the PVCBackup and source PVC.
2. Create a VolumeSnapshot for the source PVC and wait until it completes.
3. Create a snapshot PVC from the VolumeSnapshot.
4. Createa a Job to perform the restic step and wait until it completes.
5. Clean up.

It looks like this is occurring during step 3. The snapshot PVC is created with its storage resource request set to the restore size of the VolumeSnapshot created in step 2. It looks like in your case, the VolumeSnapshot restore size was 0.

Could you please run kubectl get volumesnapshot csi-lvmpv-1726406899 -o yaml and post the output here? Also, could you share some information about your setup, e.g. k8s version, which CSI is in use, and which version of it? Thanks!

[saggiyogesh]

Hello @sgielen Not doing any restore, this is after installing backsnap and it tries to take a backup.

[sgielen]

I realized that as well after placing the first comment, could you check my second comment for further followup?

[saggiyogesh]

Here is the output of kubectl get volumesnapshot csi-lvmpv-1726406899 -o yaml

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  creationTimestamp: "2024-09-15T14:41:55Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
  - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 2
  name: csi-lvmpv-1726406899
  namespace: default
  ownerReferences:
  - apiVersion: backsnap.skyb.it/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PVCBackup
    name: csi-lvmpv-1726406899
    uid: 8a03b4aa-99c7-493a-9a2a-b1582557e367
  resourceVersion: "284975"
  uid: 2c35a3eb-17dd-4f5f-9a82-ce46736d8578
spec:
  source:
    persistentVolumeClaimName: csi-lvmpv
  volumeSnapshotClassName: lvmpv-snapclass
status:
  boundVolumeSnapshotContentName: snapcontent-2c35a3eb-17dd-4f5f-9a82-ce46736d8578
  creationTime: "2024-09-15T14:41:55Z"
  readyToUse: true
  restoreSize: "0"

[sgielen]

So it seems that you're using the OpenEBS lvm-localpv provisioner, is that correct?

It seems like they don't set the status.restoreSize of VolumeSnapshots (this page also shows "RESTORESIZE: 0", and this issue does too). In this case, we could use the original size of the PVC, which is a bit wasteful (it claims unused space) but definitely better than nothing.

Although even if we do, it looks like that CSI may not support restoring from a VolumeSnapshot currently at all: openebs/lvm-localpv#13 So even if we fix this RestoreSize issue, you'll run into issues in the next step. I'm afraid this is unfixable on our side, until the CSI plugin supports restoring PVC from VolumeSnapshot (and preferably, also sets VolumeSnapshot RestoreSize at that point).

[sgielen]

@SnoozeThis openebs/lvm-localpv#13

[SnoozeThis]

(https://snoozeth.is/6H7ApEDKDsE) I will wait until openebs/lvm-localpv#13 is closed and then add a comment.