Preparing nodes for GPU passthrough

For GPU passthrough, detach the graphics card that you want to attach to VMs from the host, and then enable IOMMU. You can detach multiple graphics cards with the same VID and PID, as well as particular graphics cards by using their PCI addresses.

If you have multiple graphics cards detached from the node with pci-helper.py detach but want to use only one of them for GPU passthrough, you need to revert the detachment, and then detach one card with pci-helper.py bind-to-stub. In this case, other graphics cards on the node can be used as vGPUs.

To detach multiple graphics cards from a node and enable IOMMU

List all graphics cards on a node and obtain their VID and PID:

# lspci -nnD | grep NVIDIA
0000:01:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
0000:81:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)

[10de:1eb8] is the VID and PID of the graphics card.

Run the pci-helper.py detach script to detach all graphics cards with the same VID and PID. For NVIDIA graphics cards, additionally blacklist the Nouveau driver. For example:
```
# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py detach 10de:1eb8 --blacklist-nouveau
```
The command detaches the graphics cards with the VID and PID 10de:1eb8 from the node and prevents the Nouveau driver from loading.
Run the pci-helper.py enable-iommu script to enable IOMMU on the node:
```
# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py enable-iommu
```
The script works for both Intel and AMD processors.
Reboot the node to apply the changes:
```
# reboot
```

You can check that IOMMU is successfully enabled in the dmesg output:

# dmesg | grep -e DMAR -e IOMMU
[    0.000000] DMAR: IOMMU enabled

To revert multiple GPU detachment

In the /etc/default/grub file, locate the GRUB_CMDLINE_LINUX line, and then delete pci-stub.ids=<gpu_vid>:<gpu_pid> rd.driver.blacklist=nouveau nouveau.modeset=0. The resulting file may look as follows:
```
# cat /etc/default/grub | grep CMDLINE
GRUB_CMDLINE_LINUX="crashkernel=auto tcache.enabled=0 quiet iommu=pt"
```
Regenerate the GRUB configuration file.
- On a BIOS-based system, run:
```
# /usr/sbin/grub2-mkconfig -o /etc/grub2.cfg
```
- On a UEFI-based system, run:
```
# /usr/sbin/grub2-mkconfig -o /etc/grub2-efi.cfg
```
Delete the /etc/modprobe.d/blacklist-nouveau.conf file.
Re-create the Linux boot image by running:
```
# dracut -f
```
Reboot the node to apply the changes:
```
# reboot
```

To detach a particular graphics card from a node and enable IOMMU

List all graphics cards on a node and obtain their PCI addresses:

# lspci -nnD | grep NVIDIA
0000:01:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)
0000:81:00.0 3D controller [0302]: NVIDIA Corporation TU104GL [Tesla T4] [10de:1eb8] (rev a1)

0000:01:00.0 and 0000:81:00.0 are the PCI addresses of the graphics cards.

Run the pci-helper.py bind-to-stub script to assign the pci-stub driver to the GPU at its PCI address. For example:
```
# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py bind-to-stub 0000:01:00.0
```
The command detaches the graphics card with PCI address 0000:01:00.0 from the node and prevents the Nouveau driver from loading.
Run the pci-helper.py enable-iommu script to enable IOMMU on the node:
```
# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py enable-iommu
```
The script works for both Intel and AMD processors.
Reboot the node to apply the changes:
```
# reboot
```

You can check that IOMMU is successfully enabled in the dmesg output:

# dmesg | grep -e DMAR -e IOMMU
[    0.000000] DMAR: IOMMU enabled