Fixed by adding selinux_state: disabled to group_vars, however default in environments/common/inventory/group_vars/all/selinux.yml sets

selinux_state: permissive
selinux_policy: targeted

any reason for this?

Hmm, I saw this, even though I'm using one of the StackHPC images.

selinux is not disabled by default, hence this occurs with any unmodified cookiecutter environment regardless of image. See https://github.com/stackhpc/ansible-slurm-appliance/blob/main/environments/common/inventory/group_vars/all/selinux.yml. We do disable it in CI: https://github.com/stackhpc/ansible-slurm-appliance/blob/main/environments/.stackhpc/inventory/group_vars/selinux/overrides.yml

Prometheus should work with selinux enabled once #449 merges

Also a bit horrible that as the stackhpc env has selinux disabled, running site.yml using a default env (with it enabled) on our image means it reboots to change the status ...

Ok so the (depreciated) cloudalchemy.prometheus role is looking for the wrong packages:

TASK [cloudalchemy.prometheus : Gather variables for each operating system] ********************************************************************************************************************************************************
Friday 03 January 2025  11:11:57 +0000 (0:00:00.042)       0:09:51.507 ******** 
ok: [rl9-control] => (item=/home/rocky/slurm-app-rl9/ansible/roles/cloudalchemy.prometheus/vars/redhat.yml)

#ansible/roles/cloudalchemy.prometheus/vars/redhat.yml:
---
prometheus_selinux_packages:
  - libselinux-python
  - policycoreutils-python

but:

#ansible/roles/cloudalchemy.prometheus/vars/redhat-8.yml:
---
prometheus_selinux_packages:
  - python3-libselinux
  - python3-policycoreutils

and

[root@rl9-control rocky]# dnf list python3-libselinux
Last metadata expiration check: 0:10:20 ago on Fri 03 Jan 2025 11:03:24 AM UTC.
Installed Packages
python3-libselinux.x86_64                                                                                     3.6-1.el9                                                                                     @appstream
[root@rl9-control rocky]# dnf list python3-policycoreutils
Last metadata expiration check: 0:10:25 ago on Fri 03 Jan 2025 11:03:24 AM UTC.
Installed Packages
python3-policycoreutils.noarch   

There is no redhat-9 vars file, so basically, it doesn't support EL9 properly.

However because these come from an include_vars task, that has higher priority than any inventory, so we can't override them :-(. And we aren't using a forked role

Also explains why this works on a client with SELinux enabled (on control node), b/c they are on RockyLinux 8.