Creating virtual machines with SR-IOV network ports
-
Connect to the OpenStack command-line interface as a system administrator to authorize further OpenStack commands (refer to Connecting to OpenStack command-line interface).
# kolla-ansible post-deploy # source /etc/kolla/admin-openrc.sh
-
Create a physical compute network specifying the network adapter alias from the pci-passthrough.yaml file. For example, to create the
sriov-net
network, run:# openstack --insecure network create --provider-physical-network sriovnet --provider-network-type flat sriov-net
-
Create a subnet for the
sriov-net
network disabling the built-in DHCP server and specifying the desired IP address range. For example, to create thesriov-subnet
subnet with the 10.10.10.0/24 CIDR, run:# openstack --insecure subnet create --no-dhcp --subnet-range 10.10.10.0/24 --network sriov-net sriov-subnet
-
Create a network port in the
sriov-net
network with the direct PCI passthrough. For example, to create thesriov-port
port with the IP address 10.10.10.10 from thesriov-subnet
subnet, run:# openstack --insecure port create --network sriov-net --vnic-type=direct --fixed-ip subnet=sriov-subnet,ip-address=10.10.10.10 sriov-port
-
Create a virtual machine specifying the
sriov-port
port. Enable the--config-drive
option, to automatically assign the IP address inside the guest operating system. For example, to create thesriov-vm
from thevol1
volume and with thelarge
flavor, run:# openstack --insecure server create --port sriov-port --volume vol1 --flavor large sriov-vm --config-drive True
If the VM creation fails with the following error in /var/log/hci/nova/nova-compute.log
:
2021-08-27 17:56:21.349 6 ERROR nova.compute.manager [instance: 9fb738bf-afe5-40ef-943c- 22e43696bfd9] libvirtError: internal error: qemu unexpectedly closed the monitor: 2021-08-27T14:56:20.294985Z qemu-kvm: -device vfio-pci,host=01:00.3,id=hostdev0, bus=pci.0,addr=0x6: vfio error: 0000:01:00.3: group 1 is not viable 2021-08-27 17:56:21.349 6 ERROR nova.compute.manager [instance: 9fb738bf-afe5-40ef-943c- 22e43696bfd9] Please ensure all devices within the iommu_group are bound to their vfio bus driver.
In this case, the physical and virtual functions of the network adapter might belong to the same IOMMU group. You can check this by using the virsh nodedev-dumpxml
command and specifying the device names of physical and virtual functions. For example:
# virsh nodedev-dumpxml pci_0000_00_03_0 | grep iommuGroup <iommuGroup number='1'> </iommuGroup> # virsh nodedev-dumpxml pci_0000_00_03_1 | grep iommuGroup <iommuGroup number='1'> </iommuGroup>
The device names have the format pci_0000_<bus_number>_<device_number>_<function_number>
. These numbers can be obtained via the lspci
command:
# lspci -nn | grep Ethernet 00:03.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017] ...
In this output, 00
is the bus number, 03
is the device number, and 0
is the function number.
If the physical and virtual functions belong to the same IOMMU group, you need to detach the physical function from the node by running the pci-helper.py
script and specifying its VID and PID. For example:
# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py detach 15b3:1017