Configuring InfiniBand devices

Limitations

As the admin panel only shows IP states and does not show InfiniBand (IB) connection states, it may report plugged in but yet unconfigured IB devices as Disconnected. The status will change to Connected once you assign an IP address to such a device.
A network interface must have at least one IPv4 address.
You can only assign an IPv6 address manually. Obtaining an IPv6 address via DHCP is not supported.

Prerequisites

The Storage traffic type is moved to a dedicated network.

To configure InfiniBand devices

Admin panel

On the Infrastructure > Networks screen, assign the traffic type Storage to an empty network (without any other traffic types). If needed, create a new network by clicking Create network.
On the Infrastructure > Nodes screen, click the name of the node, go to the Network interfaces tab, and then click the network interface.
On the interface right pane, click Edit.
In the Edit network interface window, specify the network with the Storage traffic type, select Manually, and then specify the IP address in CIDR notation by clicking Add.
Specify a gateway. The provided gateway will become the node’s default.
Select Connected mode.
Enter the 65520 value in the MTU field.

Setting a custom MTU in the admin panel prior to configuring it on the network hardware will result in network failure on the node and require manual resetting. Setting an MTU that differs from the one configured on the network hardware may result in a network outage or poor performance.
Click Save to apply your changes.
Repeat the steps for each IB device on your infrastructure nodes.

Command-line interface

Use the following command:

vinfra node iface set [--ipv4 <ipv4>] [--ipv6 <ipv6>] [--gw4 <gw4>] [--gw6 <gw6>]
                      [--mtu <mtu>] [--dhcp4 | --no-dhcp4] [--dhcp6 | --no-dhcp6]
                      [--auto-routes-v4 | --ignore-auto-routes-v4]
                      [--auto-routes-v6 | --ignore-auto-routes-v6]
                      [--network <network> | --no-network] [--connected-mode | --datagram-mode]
                      [--node <node>] <iface>

--ipv4 <ipv4>: A comma-separated list of IPv4 addresses
--ipv6 <ipv6>: A comma-separated list of IPv6 addresses
--gw4 <gw4>: Gateway IPv4 address
--gw6 <gw6>: Gateway IPv6 address
--mtu <mtu>: MTU interface value
--dhcp4: Enable DHCPv4
--no-dhcp4: Disable DHCPv4
--dhcp6: Enable DHCPv6
--no-dhcp6: Disable DHCPv6
--auto-routes-v4: Enable automatic IPv4 routes
--ignore-auto-routes-v4: Ignore automatic IPv4 routes
--auto-routes-v6: Enable automatic IPv6 routes
--ignore-auto-routes-v6: Ignore automatic IPv6 routes
--network <network>: Network ID or name
--no-network: Remove a network from the interface
--connected-mode: Enable connected mode (InfiniBand interfaces only)
--datagram-mode: Enable datagram mode (InfiniBand interfaces only)
--node <node>: Node ID or hostname (default: node001.vstoragedomain)
<iface>: Network interface name

For example, to assign the network Storage to the network interface ib2 located on the node node002, enable the connected mode, set its IP address to 192.168.30.20/24, and MTU to 65520, run:

# vinfra node iface set ib2 --network Storage --node node002 --ipv4 192.168.30.20/24 \
--mtu 65520 --connected-mode