Benchmarking the network

To benchmark the infrastructure network, we test network connectivity between all of the cluster nodes, and then the overall network throughput under the maximum load.

Prerequisites

• Knowledge of the storage cluster best practices and configurations listed in Storage cluster best practices and Configuration examples.
• The iperf3 utility is installed by default or by running yum install iperf3.

To test network connectivity between nodes

1. Start the iperf3 server on each node:

   # for i in $(vinfra node list -f json | jq -r '.[].host'); do ssh $i iperf3 -s -p 30001 -D; done

2. Start the benchmark on each node:

   # NODES=`vinfra node list -f json | jq -r '.[].host' | sort | grep -v $(hostname -s)`
   # for i in $NODES; do iperf3 -c $i -p 30001 ; done | egrep -i 'local|sender|receiver'

   The output will be similar to the following:

   [  4] local 172.24.4.1 port 40666 connected to 172.24.4.2 port 30001
   [  4]   0.00-10.00  sec  28.6 GBytes  24.5 Gbits/sec    44            sender
   [  4]   0.00-10.00  sec  28.6 GBytes  24.5 Gbits/sec                  receiver
   [  4] local 172.24.4.1 port 44968 connected to 172.24.4.4 port 30001
   [  4]   0.00-10.00  sec  22.1 GBytes  19.0 Gbits/sec   1672           sender
   [  4]   0.00-10.00  sec  22.1 GBytes  19.0 Gbits/sec                  receiver
   

3. Terminate the iperf server when done.

In the test report, check the following:

• The measured network speed between all node pairs is similar, and that it is close to the rated network speed. Otherwise, this may indicate a network problem.
• The number of retries (shown in the second-to-last column) is close to zero.

To test network throughput under load

1. Start one iperf3 server on a different port per client on each node:

   # for i in $(vinfra node list -f json | jq -r '.[].host'); do ssh $i "for j in 1 2 3; \
   do iperf3 -p 3000\$j -s -D ; done" ; done

2. Start the benchmark simultaneously on all of the nodes:

   # PORT=30001 # use a different port on every client (ranged 30001…30003)
   # NODES=`vinfra node list -f json | jq -r '.[].host' | sort | grep -v $(hostname -s )`
   # for i in $NODES; do iperf3 -c $i -p $PORT ; done | egrep -i 'local|sender|receiver'

3. Terminate the iperf server when done.

The test tries to maximize network load by generating traffic between all node pairs, and then reports the measured network speed between pairs.

In the test report, check the following:

• The cumulative input and output of each node is close to the rated network speed.
• The number of retries (shown in the second-to-last column) is close to zero.