Snapshot backups with Volume Shadow Copy Service (VSS) writer

[fbellesi69]

Dear Sirs,

we are trying to backup our Domino V12.0.2 FP4 / Windows server 2019 VMs on VMWARE V7, using VEEAM V12 and the new VSS writer.
When we launch the task (LOAD BACKUPVSS -D), we see the Domino writer listed as follows:

Author Name: 'Domino Backup Writer'
Author ID: {dee12323-b0ce-4f3d-8d3a-68062f1c53cc}
Author Instance ID: {f424a567-6a23-4657-8308-8c04888bb667}
Status: [1] Stable
Last error: No errors

When we launch the backup job (application aware) on VEEAM, in the Domino log it seems that everything went successfully and the VEEAM job end without errors:

[0B10:0004-0EAC] 12/09/2024 09:55:31 BackupVSS: Domino Agent VSS Freeze Event received
[0B10:0004-0EAC] 12/09/2024 09:55:31 BackupVSS: backup command line: Binary path: [C:\Lotus\Domino\nbackup.EXE] Param: [-vss]
[13C8:0002-1024] 12/09/2024 09:55:31 Backup: Domino Database Backup
[13C8:0002-1024] 12/09/2024 09:55:31 Backup: Started
[13C8:0002-1024] 12/09/2024 09:55:31 Backup: Pruning backups
[13C8:0002-1024] 12/09/2024 09:55:31 Backup: BackupNode: [traveler01], BackupName: [default], Translog Mode: [CIRCULAR], Backup Mode: [SNAP]
[13C8:0002-1024] 12/09/2024 09:55:31 Backup: LastBackupTime: 05/09/2024 18:17:30
[13C8:0002-1024] 12/09/2024 09:55:33 Backup: Starting backup for 180 database(s) [SnapShotMode]
[13C8:0002-1024] 12/09/2024 09:55:37 Backup: Ready for VSS Snapshot
[0B10:0004-0EAC] 12/09/2024 09:55:37 BackupVSS: DominoVSSWriter::OnFreeze Domino in Freeze Mode after 4 seconds!
[2154:0002-2830] 12/09/2024 09:55:40 Router: Updating values from Configuration document
[2154:0002-2830] 12/09/2024 09:55:40 Router: notes.ini setting for Log_Mailrouting being used (note - this option may now be configured in a Server Configuration document)
[13C8:0002-1024] 12/09/2024 09:55:50 Backup: VSS Post Snapshot status reached after 11 seconds!
[13C8:0002-1024] 12/09/2024 09:55:50 Backup: Capturing changes in 180 database(s) [SnapShotMode]
[0B10:0004-0EAC] 12/09/2024 09:55:50 BackupVSS: Sending unfreeze to Domino Backup
[2154:0002-2830] 12/09/2024 09:55:51 Router: Updating values from Configuration document
[2154:0002-2830] 12/09/2024 09:55:51 Router: notes.ini setting for Log_Mailrouting being used (note - this option may now be configured in a Server Configuration document)
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: VSS Snapshot Backup finalized
[0B10:0004-0EAC] 12/09/2024 09:56:28 BackupVSS: DominoVSSWriter::OnPostSnapshot Post backup changes applied after 34 seconds.
[13C8:0002-1024] 12/09/2024 09:56:28 Backup:
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: --- Backup Summary ---
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Previous Backup : 05/09/2024 18:17:30
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Start Time : 12/09/2024 09:55:31
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: End Time : 12/09/2024 09:56:28
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Runtime : 00:00:56.203
[13C8:0002-1024] 12/09/2024 09:56:28 Backup:
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: All : 180
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Processed : 180
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Excluded : 0
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Pending Compact : 0
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Compact Retries : 0
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Backup Errors : 0
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Not Modified : 0
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Delta Files : 5
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Delta applied : 0
[13C8:0002-1024] 12/09/2024 09:56:28 Backup:
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Total DB Size : 3,5 GB
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Total DeltaSize : 335,1 KB
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Data Rate : 63,5 MB/sec
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: --- Backup Summary ---
[13C8:0002-1024] 12/09/2024 09:56:28 Backup:
[13C8:0002-1024] 12/09/2024 09:56:28 Backup: Finished

But if we analyze the logs in the dominobackup.nsf db, we see that the document in the view has the hourglass icon and in the log document the "Snapshot completed" field is empty (see attached screenshots).

Additionally, the status of the writer is as follows:

Author Name: 'Domino Backup Writer'
Author ID: {dee12323-b0ce-4f3d-8d3a-68062f1c53cc}
Author Instance ID: {f424a567-6a23-4657-8308-8c04888bb667}
Status: [11] Failed
Last error: No errors

Furthermore, if we shutdown the DOMINOVSS task we see the following errors in the output of the operation:

BackupVSS: DominoVSSWriter::DominoVSSWriter called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnPrepareBackup called
BackupVSS: DominoVSSWriter::OnPrepareSnapshot called
BackupVSS: DominoVSSWriter::OnFreeze called
BackupVSS: Requested backup operation done
BackupVSS: DominoVSSWriter::OnThaw called
BackupVSS: DominoVSSWriter::OnPostSnapshot called
BackupVSS: OnPostSnapshot - Backup type: FULL, Backup level: FRONT_END
BackupVSS: VSS-Error getting snapshot device name: 0x80042308
BackupVSS: Writable VSS Snapshot not supported - Not performing auto recovery.
BackupVSS: Requested backup operation done
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnBackupShutdown called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::OnIdentify called
BackupVSS: DominoVSSWriter::Uninitialize called
BackupVSS: DominoVSSWriter::~DominoVSSWriter called
BackupVSS: DominoVSSWriter::Uninitialize called
[0B10:0002-0348] 12/09/2024 10:06:41 BackupVSS: Domino Backup VSS plug-in shutdown

Please help...
Francesco Bellesi

[Daniel-Nashed]

We are working with Veeam on a solution. This isn't a Domino bug and the logging has been improved in Domino 14.0 FP2 to show more details.

The actual problem is that the standard Veeam configuration for application consistency only causes a Snapshot of the first disk C:. If Domino databases are located on another disk (which is usually the case), no VSS Snapshot is currently created.

There are two different work-arounds for now.

1. Use a Veeam local backup agent

The easy work-around which will always work on any Veeam environment also without VMware backend, is to install the Veeam Windows agent and configure Backup locally on the machine.
This will generate the right VSS snapshots and also allows you to select what to backup in more detail.

2. Use VMware Vsphere VM snapshots

The other solution currently is to use a native VMware snapshot.
This will only work with a virtual machine where the disks to snapshot are configured as dependent disks.
Because only dependent disks are snapshotted along with the VM.

In addition VMware tools need to be installed on the machine (which should be the case on all Windows machines).

For this type of configuration the standard configuration for application aware processing needs to be disabled and the VMware snapshot configuration needs to be enabled.

1. Disable Enable application-aware processing on Guest Processing tab of the backup job.
2. On the Storage tab select Advanced...
3. Open the vSphere tab
4. Select Enable VMware Tools quiescence
5. Enable all Change blog tracking options below on the same screen (should be already enabled in most cases)

On the VMware side make sure the disk is a dependent disk.
In addition make sure the setting Quiesce guest file system (needs VMware tools installed). needs to be selected in the VM configuration.

This configuration will switch to VSphere snapshots instead of Veeam enabled snapshots.
We are actively working with Veeam to find a solution. This will probably be a fix on the Veeam side.
For now you only have the described options.

---

For the Domino 14.0 code stream there is a known issue in 14.0 FP1 where the snapshot volume check is case sensitive.
This could also cause issues and we recommend updating to Domino 14.0 FP2IF1.
But this is very likely not the root cause of your problem. If you are using the standard configuration for a Veeam backup you are running into the described issue.

Domino 12.0.2 does not have the fix and will not check which disk is snapshotted.
This enhancement has been only introduced in 14.0.

See Issue: Windows server backup trigger domino VSS writer even when domino data directory not in scope for backup
for details.

Please let us know if this description helps and which of the two options you might want to use + if this solves your problem.

Thanks

Daniel

[fbellesi69]

Dear Daniel, thanks a lot for your clarification. I will check with the customer if the second workaround is suitable...
Have a nice day
Francesco Bellesi

[Daniel-Nashed]

Let me know if you need anything. This is complicated and solutions like this need multiple teams to work together.
The larger the company the more separate teams involved

• Domino
• Windows
• VMware
• Veeam

could be all in different team's responsibility.
It's much easier when you are the admin for all components like in my case.

You have a ticket open. If there is any customer sensitive information you can't share here, please provide it thru the ticket.

But for anything that can be shared here (maybe with some manual obfuscation of customer information) the GitHub issue is the much faster communication.

For backup integration related issues, this GitHub project is the recommended way.

[Daniel-Nashed]

I changed the issue to not be a bug. I don't think we have a bug on the Domino side here as explained. But that does not change anything on the severity of the issue you are facing nor my priority in helping you!
We might need to document this also in the GitHub repository.

[fbellesi69]

Dear Daniel,
the customer evaluated both proposed workarounds, and decided not to apply them. It is preferable to wait until the integration work you are carrying out with VEEAM is completed so that you can use the application aware mode, also used for backing up other applications.
Thank you for your precious support and good work!

[Daniel-Nashed]

Thanks for your nice feedback and your info!
There is no integration work open. Starting with Domino 12.0.1 VSS Writer with Auto recovery snapshots is the preferred integration approach. For the backup side nothing is usually needed to be configured. The VSS Writer integration is taking care of it.

Only for restore you need an integration. But with Auto Recovery snapshots the data is consistent in your VSS triggered backup.

The current challenge we are facing is a limitation on the Veeam side for the standard backup configuration.
The error was masked because Domino did not check for the snapshot drives.
We had the impression registering for the right components would only trigger the Domino VSS Writer when we are in scope.

But it turned out we really had to check explicitly for the volumes requested. Implemented in 12.0.1 FP1 which revealed we have not seen before.

We have to wait for Veeam, but there isn't any code change pending on Domino Backup side.

Sorry to hear the two other backup options don't work for you as a interim solution.
I personally think having the Veeam Windows Backup agent installed is a good option.
But yes it might not fit into all environments.

By the way other major vendors also have challenges with VSS Snapshots. They don't trigger a Auto Recovery snapshot.
But they are also working on a solution right now.

VSS technology isn't new. But the devil is in the detail.