[BUG] Minion upgrade to 3006.9 fails on Windows with "specified service already exists" error

[jmcmillan89]

Description
After upgrading our Salt master to 3006.9, we have turned our attention to getting our minions upgraded to 3006.9 as well. We are having difficulties with our Windows minions, as the silent install/upgrade does not complete successfully. When upgrading manually, it errors out at the very end of the installation because the salt-minion service already exists. Just to be clear, when I say upgrade, I mean that we already have the Salt minion installed and running at an older version, and are just installing from a newer Salt minion binary (this has worked for us in the past).

Apologies for submitting this as a bug, but I couldn't find a more appropriate type.

Setup
We have many Windows machines in our organization running various versions of Windows Server -- some as old as 2012 (very few) but most are 2016 or 2019. All of our minions are currently on version 3004.2, and we are attempting a silent upgrade using the following simple Salt state:

Install salt minion 3006.9:
  file.managed:
    - source: https://my-company-artifactory/saltproject-generic-remote/windows/3006.9/Salt-Minion-3006.9-Py3-AMD64-Setup.exe
    - name: "C:\\Windows\\Temp\\Salt-Minion-3006.9-Py3-AMD64-Setup.exe"
    - skip_verify: True
  cmd.run:
    - name: "C:\\Windows\\Temp\\Salt-Minion-3006.9-Py3-AMD64-Setup.exe /S"

And applying with salt -L my-minion.foo.bar state.apply minion.upgrade. For reference, this is the exact same state we used to upgrade our Windows minions to 3004.2, obviously using that binary. It is expected that this state will fail, as the minion restarts during the installation.

After applying this state, if I do a salt -L my-minion.foo.bar grains.get saltversion it is still on 3004.2 -- if I hop onto the machine, and look into C:\Program Files\Salt Project\Salt I can see that the directory structure has been updated -- e.g. there is salt-call.exe instead of salt-call.bat -- but there are also still relics from the old 3004.2 installation, including the bin/ directory:

If I inspect the salt-minion service, I can see that it is still pointed to C:\Program Files\Salt Project\Salt\bin\ssm.exe instead of C:\Program Files\Salt Project\Salt\salt-minion.exe like I would expect:

To my eyes, it looks as though the minion is in a "half-state" between 3004.2 and 3006.9. So, I uninstalled the Salt minion, re-installed at 3004.2, then manually did an upgrade to 3006.9 (basically, just run the .exe and click through the screens). It almost got to the end of the install when it failed with the following error:

It would appear that this binary cannot handle the scenario where the salt-minion service already exists -- in my experience, this was not the case when upgrading to 3004.2. As a result, we are kind of stuck automating the upgrade of our Windows minions as I cannot find a way to workaround this.

For this testing I am using an on-prem machine, and I have also tried using my own laptop and gotten the same result.

Expected behavior
I would expect the Salt-Minion-3006.9-Py3-AMD64-Setup.exe binary to be able to handle the use-case where the salt-minion service already exists, and upgrade/re-configure it accordingly.

One workaround I tried was to script the upgrade by executing a PowerShell script with cmd.run in the background -- so basically, the master will kick off this script and then return immediately, not waiting for it to complete:

# Perform a silent install of the new minion binary
$x = Start-Process -FilePath "C:\Windows\Temp\Salt-Minion-3006.9-Py3-AMD64-Setup.exe" -ArgumentList @("/S") -Wait -PassThru -NoNewWindow

# This does upgrade the minion, HOWEVER it fails to update the SERVICE b/c the service already exists
# These commands set the service properties to what they would be if we did a CLEAN/FRESH install of 3006.9
$x = Start-Process -FilePath "C:\Program Files\Salt Project\Salt\ssm.exe" -ArgumentList @("set salt-minion Application C:\Program Files\Salt Project\Salt\salt-minion.exe") -Wait -PassThru -NoNewWindow
$x = Start-Process -FilePath "C:\Program Files\Salt Project\Salt\ssm.exe" -ArgumentList @('set salt-minion AppParameters -c "C:\ProgramData\Salt Project\Salt\conf" -l quiet') -Wait -PassThru -NoNewWindow
$x = Start-Process -FilePath "C:\Program Files\Salt Project\Salt\ssm.exe" -ArgumentList @("set salt-minion AppDirectory C:\Program Files\Salt Project\Salt") -Wait -PassThru -NoNewWindow

# Stop the minion
$x = Start-Process -FilePath "C:\Program Files\Salt Project\Salt\ssm.exe" -ArgumentList @("stop salt-minion") -Wait -PassThru -NoNewWindow

# Delete the old bin directory
# This is leftovers from the old minion (3004.2) and the service was pointing to files in here
# Now that we have updated the service to use the new files, this directory can be removed
Remove-Item "C:\Program Files\Salt Project\Salt\bin" -Force -Recurse

# Start the minion
$x = Start-Process -FilePath "C:\Program Files\Salt Project\Salt\ssm.exe" -ArgumentList @("start salt-minion") -Wait -PassThru -NoNewWindow

This is not working for me either -- my ssm.exe commands to edit the service configuration are not taking effect, and the script abruptly stops as soon as the salt-minion service is stopped or restarted (I have a hunch that even though the script is running in the background, its parent process is still the Salt minion service, so killing the service kills the script). This is also editing the service while that same service is running, so I don't even know if this is a legitimate approach -- either way, it's not working and as you can see, I'm getting desperate.

Is there a recommended or more graceful way to silently upgrade Windows minions? We simply have too many to upgrade them manually.