Creating virtual machines with SR-IOV network ports

Limitations

Virtual machines with attached PCI devices cannot be live migrated.

Prerequisites

The compute cluster is reconfigured for PCI passthrough, as described in Enabling PCI passthrough and vGPU support.
To authorize further OpenStack commands, the OpenStack command-line client must be configured, as outlined in Connecting to OpenStack command-line interface.

To create a virtual machine with an SR-IOV network port

Create a physical compute network specifying the network adapter alias from the pci-passthrough.yaml file and the default vNIC type direct. You also need to disable the built-in DHCP server and specify the desired IP address range. For example, to create the sriov-network network with the 10.10.10.0/24 CIDR, run:
```
# vinfra service compute network create sriov-network --physical-network sriovnet --default-vnic-type direct \
--no-dhcp --cidr 10.10.10.0/24
```
Create a virtual machine specifying the new network. For example, to create the VM sriov-vm from the template centos7 and with the large flavor, run:
```
# vinfra service compute server create sriov-vm --network id=sriov-network --volume source=image,size=11,id=centos7 --flavor large
```

If the VM creation fails

Check whether the following error appears in /var/log/hci/nova/nova-compute.log:

2021-08-27 17:56:21.349 6 ERROR nova.compute.manager [instance: 9fb738bf-afe5-40ef-943c-
22e43696bfd9] libvirtError: internal error: qemu unexpectedly closed the monitor: 
2021-08-27T14:56:20.294985Z qemu-kvm: -device vfio-pci,host=01:00.3,id=hostdev0,
bus=pci.0,addr=0x6: vfio error: 0000:01:00.3: group 1 is not viable
2021-08-27 17:56:21.349 6 ERROR nova.compute.manager [instance: 9fb738bf-afe5-40ef-943c-
22e43696bfd9] Please ensure all devices within the iommu_group are bound to their vfio 
bus driver.

In this case, the physical and virtual functions of the network adapter might belong to the same IOMMU group. You can check this by using the virsh nodedev-dumpxml command and specifying the device names of physical and virtual functions. For example:

# virsh nodedev-dumpxml pci_0000_00_03_0 | grep iommuGroup
    <iommuGroup number='1'>
    </iommuGroup>
# virsh nodedev-dumpxml pci_0000_00_03_1 | grep iommuGroup
    <iommuGroup number='1'>
    </iommuGroup>

The device names have the format pci_0000_<bus_number>_<device_number>_<function_number>. These numbers can be obtained via the lspci command:

# lspci -nn | grep Ethernet
00:03.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
...

In this output, 00 is the bus number, 03 is the device number, and 0 is the function number.

If the physical and virtual functions belong to the same IOMMU group, you need to detach the physical function from the node by running the pci-helper.py script and specifying its VID and PID. For example:

# /usr/libexec/vstorage-ui-agent/bin/pci-helper.py detach 15b3:1017