Creating encrypted backups from encrypted ZFS pools

[LukasKnuth]

I've created an encrypted ZFS pool with the goal of having the volume snapshots stored in S3 be encrypted. The snapshot is taken as expected, but it's not encrypted.

I posted this originally over at openebs/velero-plugin#181 but it hasn't gotten any attention yet.

I'm posting it here now because I think the Snapshot behavior is actually implemented in this repository and some insight could be given by someone more knowledgeable than me.

---

What steps did you take and what happened:

I'm using OpenEBS ZFS-localPV

1. Added a new zpool with sudo zpool create -o ashift=12 -o feature@encryption=enabled -O encryption=on -O keylocation=file:///root/zfs-encrypt.key -O keyformat=raw encrypted-pool `sudo losetup -f /tmp/zfs-encrypted.img --show`
2. Created a new StorageClass to create PVCs for this pool
3. Setup a new PVC from the storage class and wrote some plain data into it
4. Ran a Velero backup velero backup create encrypted-test --snapshot-volumes --include-namespaces=apps --volume-snapshot-locations=default --storage-location=default
5. The backup completed successfully and the data is found on my S3 storage
6. Downloaded the zfs-pvc-0828badb-1386-4869-a475-00f9795d262d-encrypted-test file from the S3 bucket (UUID matches my PVC on the cluster)
7. Ran strings zfs-pvc-0828badb-1386-4869-a475-00f9795d262d-encrypted-test | grep find_me and found the contents of the file on the encrytped PVC

What did you expect to happen:

The strings command doesn't print the contents of the file backed up from the encrypted pool.

The output of the following commands will help us better understand what's going on:

$ kubectl get storageclass/openebs-zfs-encrypted -o yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-zfs-encrypted
  uid: 6a79fea8-7bcc-4ea0-a609-162b0489a25c
parameters:
  dedup: "off"
  fstype: zfs
  poolname: encrypted-pool
provisioner: zfs.csi.openebs.io
reclaimPolicy: Delete
volumeBindingMode: Immediate

$ zfs get -p encryption,keystatus encrypted-pool
NAME            PROPERTY    VALUE        SOURCE
encrypted-pool  encryption  aes-256-gcm  -
encrypted-pool  keystatus   available    -

$ zfs get -p encryption,keystatus encrypted-pool/pvc-0828badb-1386-4869-a475-00f9795d262d@encrypted-test
NAME                                                                    PROPERTY    VALUE        SOURCE
encrypted-pool/pvc-0828badb-1386-4869-a475-00f9795d262d@encrypted-test  encryption  aes-256-gcm  -
encrypted-pool/pvc-0828badb-1386-4869-a475-00f9795d262d@encrypted-test  keystatus   available    -

$ kubectl -n apps get pvc/encrypted-storage
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            AGE
encrypted-storage   Bound    pvc-0828badb-1386-4869-a475-00f9795d262d   1Gi        RWO            openebs-zfs-encrypted   53m

Anything else you would like to add:

Since there is no specific documentation on this subject in either this nor the drivers repository, I'm not sure if I might just have misunderstood or misconfigured something.

What I'm trying to do is have both encrypted ZFS filesystems backing my PVCs on the actual disk AND have the backup be encrypted in the cloud as well! Meaning it's not necessarily possible to restore a backup (fully) without the encryption key from the host (specified when creating the zpool) and the data on my PVCs is encrypted at rest.

Environment:

• Velero version (use velero version): 1.9.0
• Velero features (use velero client config get features): NOT SET
• Velero-plugin version: 3.3.0
• OpenEBS version: 2.1.0
• Kubernetes version (use kubectl version): v1.23.6
• Kubernetes installer & version: v1.24.3+k3s1
• Cloud provider or hardware configuration: Raspberry Pi 4
• OS (e.g. from /etc/os-release): Ubuntu 20

[LukasKnuth]

I have done some more digging on how this works internally and have come to understand the following:

1. When the Velero plugin creates a backup, it will initialize a ZFS plugin https://github.dev/openebs/velero-plugin/blob/cea57783e3ed887d2b7b0e7bafc436ff26bd9a7b/pkg/zfs/plugin/zfs.go#L82
2. This instance holds config for the current backup process, which includes a "remote destination address", the first IPv4 loopback address on the system: https://github.dev/openebs/velero-plugin/blob/cea57783e3ed887d2b7b0e7bafc436ff26bd9a7b/pkg/zfs/utils/utils.go#L35
3. This leads to creating a ZFSBackup resource in Kubernetes https://github.dev/openebs/velero-plugin/blob/cea57783e3ed887d2b7b0e7bafc436ff26bd9a7b/pkg/zfs/plugin/backup.go#L125 that is in term picked up by the OpenEBS operator and executed https://github.dev/openebs/zfs-localpv/blob/8d2a24e29b6089d715d18870a093bd8702e91199/pkg/mgmt/backup/backup.go#L88
4. The call to CreateBackup ultimately builds a command that is exec'ed in a bash environment https://github.dev/openebs/zfs-localpv/blob/8d2a24e29b6089d715d18870a093bd8702e91199/pkg/zfs/zfs_util.go#L308 which ends up being bash -c zfs send <pool>/<mount>@<snapshot_name> | nc -w 3 127.0.0.1:9011

• The IP is the loopback address from step 1
• The port is provided by the velero plugin but is hard-coded: https://github.dev/openebs/velero-plugin/blob/develop/pkg/zfs/plugin/zfs.go#L53
• nc is the net-cat utility, a program to send and receive arbitrary data. The zfs send command writes the actual snapshot data to stdout and nc then sends it over the network "as is"

5. Back in the velero plugin, we sping up a server and listen for the zfs send output to pipe it to S3: https://github.dev/openebs/velero-plugin/blob/cea57783e3ed887d2b7b0e7bafc436ff26bd9a7b/pkg/clouduploader/operation.go#L50

So far so good, the critical point here is in step 4 above. To quote from the zfs send manpage (emphasis mine):

-w, --raw
For encrypted datasets, send data exactly as it exists on disk. This allows backups to be taken even if encryption
keys are not currently loaded. The backup may then be received on an untrusted machine since that machine will not
have the encryption keys to read the protected data or alter it without being detected. Upon being received, the
dataset will have the same encryption keys as it did on the send side, although the keylocation property will be
defaulted to prompt if not otherwise provided. For unencrypted datasets, this flag will be equivalent to -Lec.
Note that if you do not use this flag for sending encrypted datasets, data will be sent unencrypted and may be re-
encrypted with a different encryption key on the receiving system, which will disable the ability to do a raw send
to that system for incrementals.

I have tested this locally by manually creating a ZFS snapshot of a PVC which has a file called flag.txt with the content find_me on it:

$ zfs snapshot encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895@testing
$ zfs get -p encryption,keystatus
NAME                                                             PROPERTY    VALUE        SOURCE
encrypted-pool                                                   encryption  aes-256-gcm  -
encrypted-pool                                                   keystatus   available    -
encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895          encryption  aes-256-gcm  -
encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895          keystatus   available    -
encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895@testing  encryption  aes-256-gcm  -
encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895@testing  keystatus   available    -

$ zfs send -w encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895@testing > snapshot.dat
$ strings snapshot.dat | grep find_me
$ strings snapshot.dat | grep flag.txt
# NOTE: No output meaning the text wasn't found in the blob

$ zfs send encrypted-pool/pvc-aa201364-2391-4e68-bf18-84b625aa6895@testing > no_w.dat
$ strings no_w.dat | grep find_me
find_me
$ strings no_w.dat | grep flag.txt
flag.txt

My understanding of this behavior is that the omission of the -w flag causes the zfs send command to decrypt the content of the pvc, since the encryption key is still loaded. This is the exact opposite of the behavior I was intending to get. This is "fixed" by simply adding the missing flag and sending the data raw "as is" with encryption in tact, however I don't know the possible side-effects a change like this could have. Will investigate...

[avishnu]

Thanks for the extensive debugging and providing a possible solution. Please feel free to contribute with a PR on the https://github.com/openebs/zfs-localpv repository.