Creating virtual machines with SR-IOV network ports

Limitations

Virtual machines with attached PCI devices cannot be live migrated.

Prerequisites

The compute cluster is reconfigured for PCI passthrough, as described in Enabling PCI passthrough and vGPU support.
To authorize further OpenStack commands, the OpenStack command-line client must be configured, as outlined in Connecting to OpenStack command-line interface.

To create a virtual machine with an SR-IOV network port

Create a physical compute network specifying the network adapter alias from the pci-passthrough.yaml file. For example, to create the sriov-net network, run:
```
# openstack --insecure network create --provider-physical-network sriovnet --provider-network-type flat sriov-net
```
Create a subnet for the sriov-net network disabling the built-in DHCP server and specifying the desired IP address range. For example, to create the sriov-subnet subnet with the 10.10.10.0/24 CIDR, run:
```
# openstack --insecure subnet create --no-dhcp --subnet-range 10.10.10.0/24 --network sriov-net sriov-subnet
```
Create a network port in the sriov-net network with the direct PCI passthrough. For example, to create the sriov-port port with the IP address 10.10.10.10 from the sriov-subnet subnet, run:
```
# openstack --insecure port create --network sriov-net --vnic-type=direct --fixed-ip subnet=sriov-subnet,ip-address=10.10.10.10 sriov-port
```
Create a virtual machine specifying the sriov-port port. Enable the --config-drive option, to automatically assign the IP address inside the guest operating system. For example, to create the sriov-vm from the vol1 volume and with the large flavor, run:
```
# openstack --insecure server create --port sriov-port --volume vol1 --flavor large sriov-vm --config-drive True
```

If the VM creation fails

Check whether the following error appears in /var/log/hci/nova/nova-compute.log:

2021-08-27 17:56:21.349 6 ERROR nova.compute.manager [instance: 9fb738bf-afe5-40ef-943c-
22e43696bfd9] libvirtError: internal error: qemu unexpectedly closed the monitor: 
2021-08-27T14:56:20.294985Z qemu-kvm: -device vfio-pci,host=01:00.3,id=hostdev0,
bus=pci.0,addr=0x6: vfio error: 0000:01:00.3: group 1 is not viable
2021-08-27 17:56:21.349 6 ERROR nova.compute.manager [instance: 9fb738bf-afe5-40ef-943c-
22e43696bfd9] Please ensure all devices within the iommu_group are bound to their vfio 
bus driver.

In this case, the physical and virtual functions of the network adapter might belong to the same IOMMU group. You can check this by using the virsh nodedev-dumpxml command and specifying the device names of physical and virtual functions. For example:

# virsh nodedev-dumpxml pci_0000_00_03_0 | grep iommuGroup
    <iommuGroup number='1'>
    </iommuGroup>
# virsh nodedev-dumpxml pci_0000_00_03_1 | grep iommuGroup
    <iommuGroup number='1'>
    </iommuGroup>

The device names have the format pci_0000_<bus_number>_<device_number>_<function_number>. These numbers can be obtained via the lspci command:

# lspci -nn | grep Ethernet
00:03.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
...

In this output, 00 is the bus number, 03 is the device number, and 0 is the function number.

If the physical and virtual functions belong to the same IOMMU group, you need to detach the physical function from the node by running the pci-helper.py script and specifying its VID and PID. For example:

# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py detach 15b3:1017