Network recommendations

Recommendations for network hardware

Network latency dramatically reduces cluster performance. Use quality network equipment with low latency links. Do not use consumer-grade network switches.
Do not use desktop network adapters like Intel EXPI9301CTBLK or Realtek 8129 as they are not designed for heavy load and may not support full-duplex links. Also use non-blocking Ethernet switches.
Only Mellanox ConnectX-4 and ConnectX-5 InfiniBand adapters are supported for the RDMA mode.
If you use Mellanox adapters and AMD Epyc Rome CPU on nodes, ensure that SR-IOV is enabled in BIOS. Otherwise, this may lead to data loss.
Adapters using the BNX2X driver, such as Broadcom Limited BCM57840 NetXtreme II 10/20-Gigabit Ethernet / HPE FlexFabric 10Gb 2-port 536FLB Adapter, are not recommended. They limit MTU to 3616, which affects the cluster performance.
Enabling RDMA requires compute and storage networks to be physically separated. That means you should have separate network ports for the compute and storage traffic. If you use the recommended approach with bonded network ports, you should have two network ports for the storage network and two network ports for the compute network. To learn how to use a compute trunk network, refer to Connecting virtual switches to trunk interfaces.

Recommendations for network security

Use separate networks (and, ideally albeit optionally, separate network adapters) for internal and public traffic. Doing so will prevent public traffic from affecting cluster I/O performance and also prevent possible denial-of-service attacks from the outside.
To avoid intrusions, Virtuozzo Hybrid Infrastructure should be on a dedicated internal network inaccessible from outside.
Even though cluster nodes have the necessary iptables rules configured, we recommend using an external firewall for untrusted public networks, such as the Internet.

Recommendations for network performance

Use one 1 Gbit/s link per each two HDDs on the node (rounded up). For one or two HDDs on a node, two bonded network interfaces are still recommended for high network availability. The reason for this recommendation is that 1 Gbit/s Ethernet networks can deliver 110-120 MB/s of throughput, which is close to sequential I/O performance of a single disk. Since several disks on a server can deliver higher throughput than a single 1 Gbit/s Ethernet link, networking may become a bottleneck.
For maximum sequential I/O performance, use one 1 Gbit/s link per each hard drive or one 10 Gbit/s link per node. Even though I/O operations are most often random in real-life scenarios, sequential I/O is important in backup scenarios.
For maximum overall performance, use one 10 Gbit/s link per node (or two bonded for high network availability).
It is not recommended to configure 1 Gbit/s network adapters to use non-default MTUs (for example, 9000-byte jumbo frames). Such settings require additional configuration of switches and often lead to human error. 10+ Gbit/s network adapters, on the other hand, need to be configured to use jumbo frames to achieve full performance. You will need to configure the same MTU value on each router and switch on the network (refer to your network equipment manuals), as well as on each node’s network card, bond, or VLAN. The MTU value is set to 1500 by default.

Network recommendations for clients

The following table lists the maximum network performance a client can get with the specified network interface. The recommendation for clients is to use 10 Gbps network hardware between any two cluster nodes and minimize network latencies, especially if SSD disks are used.

Maximum client network performance
Storage network interface	Node max. I/O	VM max. I/O (replication)	VM max. I/O (erasure coding)
1 Gbps	100 MB/s	100 MB/s	70 MB/s
2 x 1 Gbps	~175 MB/s	100 MB/s	~130 MB/s
3 x 1 Gbps	~250 MB/s	100 MB/s	~180 MB/s
10 Gbps	1 GB/s	1 GB/s	700 MB/s
2 x 10 Gbps	1.75 GB/s	1 GB/s	1.3 GB/s