Preparing nodes for SR-IOV
To use SR-IOV capabilities, check that the network adapter that you want to pass through supports them, and then enable IOMMU. For Mellanox network adapters, you need to additionally enable SR-IOV in firmware.
To check that a network adapter supports SR-IOV
List all network adapters on a node and obtain their VID and PID:
# lspci -nnD | grep Ethernet 0000:00:03.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017] 0000:00:04.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
[15b3:1017]
is the VID and PID of the network adapter.
Check that the chosen network adapter supports SR-IOV by using its VID and PID:
# lspci -vv -d 15b3:1017 | grep SR-IOV Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV) Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
To enable IOMMU on a node
Run the pci-helper.py
script, and then reboot the node to apply the changes:
# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py enable-iommu # reboot
The script works for both Intel and AMD processors.
You can check that IOMMU is successfully enabled in the dmesg
output:
# dmesg | grep -e DMAR -e IOMMU [ 0.000000] DMAR: IOMMU enabled
To enable SR-IOV in firmware for Mellanox network adapters
-
Download Mellanox Firmware Tools (MFT) from the official website and extract the archive on the node. For example:
# wget https://www.mellanox.com/downloads/MFT/mft-4.17.0-106-x86_64-rpm.tgz # tar -xvzf mft-4.17.0-106-x86_64-rpm.tgz
-
Install the package, and then start Mellanox Software Tools (MST):
# yum install rpm-build # . mft-4.17.0-106-x86_64-rpm/install.sh # mst start
-
Determine the MST device path:
# mst status
-
Query the current configuration:
# mlxconfig -d /dev/mst/mt4119_pciconf0 q ... Configurations: ... NUM_OF_VFS 4 # Number of activated VFs SRIOV_EN True(1) # SR-IOV is enabled ...
-
Set the desired values, if necessary. For example, to increase the number of virtual functions to 8, run:
# mlxconfig -d /dev/mst/mt4119_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=8
- Reboot the node to apply the changes.