Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vgpu: Improve reliability of enabling sriov #37

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

vgpu: Improve reliability of enabling sriov #37

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2408b38
Make enabling sriov more reliable
jovial Jan 10, 2025
aa10d8a
Add retries to mdev unit
jovial Jan 10, 2025
486a47e
Take2: Fix failure to start on dependency failure
jovial Jan 10, 2025
b75712d
Improve comment
jovial Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions roles/vgpu/templates/nvidia-mdev.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,19 @@ Before=docker.service
{% if vgpu_definition.mig_devices is defined %}
After=nvidia-mig-manager.service
Requires=nvidia-mig-manager.service
{% else %}
After=nvidia-sriov-{{ vgpu_definition.pci_address }}.service
Requires=nvidia-sriov-{{ vgpu_definition.pci_address }}.service
{% endif %}

[Service]
Restart=on-failure
RestartSec=30
Type=oneshot
User=root
ExecStartPre=/bin/sleep 5
{% if vgpu_definition.mig_devices is not defined %}
# Workaround lack of UpheldBy/RestartMode=direct in systemd<254 to ensure unit is
# started when the dependency fails, see:
# https://unix.stackexchange.com/questions/213185/restarting-systemd-service-on-dependency-failure
ExecStartPre=/usr/bin/systemctl is-active nvidia-sriov-{{ vgpu_definition.pci_address }}.service
{% endif %}
ExecStart=-/usr/sbin/mdevctl start --uuid %i
RemainAfterExit=yes

Expand Down
3 changes: 3 additions & 0 deletions roles/vgpu/templates/nvidia-sriov.service.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,15 @@ After=local-fs.target {{ vgpu_systemd_device[vgpu_definition.pci_address] }}
Wants={{ vgpu_systemd_device[vgpu_definition.pci_address] }}

[Service]
Restart=on-failure
RestartSec=30
Type=oneshot
User=root
# NOTE(wszumski): There is a race in the driver initialization where if we run this too early, then
# the mdev_support_devices entry doesn't show up in sysfs. I was unable to get this to show up again
# without a reboot.
ExecStartPre=/bin/sleep 5
ExecStart=/bin/bash -c "echo '{{ vgpu_definition.pci_address }}' > /sys/bus/pci/drivers/nvidia/bind || true"
ExecStart=/usr/lib/nvidia/sriov-manage -e {{ vgpu_definition.pci_address }}
RemainAfterExit=yes

Expand Down