Retry NodeConfig mounts

[rzetelskik]

What should the feature do?

When a mount fails, say because the actual filesystem needed a repair, we won't try again. Well systemd doesn't :(

Say when the admin runs the notorious xfs_repair he has no idea there is a unit 'mnt-persistent\x2dvolumes.mount' that needs to be restarted. With some luck he will reboot the node...

What is the use case behind this feature?

Stability

Anything else we need to know?

root@ubuntu-2204:~# systemctl status 'mnt-persistent\x2dvolumes.mount'
× mnt-persistent\x2dvolumes.mount - Managed mount by Scylla Operator
     Loaded: loaded (/etc/systemd/system/mnt-persistent\x2dvolumes.mount; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-11-10 18:50:29 UTC; 10min ago
      Where: /mnt/persistent-volumes
       What: /dev/md127
        CPU: 5ms

Nov 10 18:50:29 ubuntu-2204 systemd[1]: Mounting Managed mount by Scylla Operator...
Nov 10 18:50:29 ubuntu-2204 systemd[1]: mnt-persistent\x2dvolumes.mount: Mount process exited, code=exited, status=32/n/a
Nov 10 18:50:30 ubuntu-2204 mount[3469]: mount: /mnt/persistent-volumes: mount(2) system call failed: Structure needs cleaning.
Nov 10 18:50:29 ubuntu-2204 systemd[1]: mnt-persistent\x2dvolumes.mount: Failed with result 'exit-code'.
Nov 10 18:50:29 ubuntu-2204 systemd[1]: Failed to mount Managed mount by Scylla Operator.
root@ubuntu-2204:~# systemctl restart 'mnt-persistent\x2dvolumes.mount'
root@ubuntu-2204:~# systemctl status 'mnt-persistent\x2dvolumes.mount'
● mnt-persistent\x2dvolumes.mount - Managed mount by Scylla Operator
     Loaded: loaded (/etc/systemd/system/mnt-persistent\x2dvolumes.mount; enabled; vendor preset: enabled)
     Active: active (mounted) since Fri 2023-11-10 19:01:32 UTC; 1s ago
      Where: /mnt/persistent-volumes
       What: /dev/md127
      Tasks: 0 (limit: 4535)
     Memory: 12.0K
        CPU: 8ms
     CGroup: /system.slice/mnt-persistent\x2dvolumes.mount

Nov 10 19:01:32 ubuntu-2204 systemd[1]: Mounting Managed mount by Scylla Operator...
Nov 10 19:01:32 ubuntu-2204 systemd[1]: Mounted Managed mount by Scylla Operator.
root@ubuntu-2204:~# 

Ref #1559

/kind feature
/priority important-longterm
/triage accepted
/assign