Benchmarking the network
To benchmark the infrastructure network, we test network connectivity between all of the cluster nodes, and then the overall network throughput under the maximum load.
Prerequisites
- Knowledge of the storage cluster best practices and configurations listed in Storage cluster best practices and Configuration examples.
- The
iperf3
utility is installed by default or by runningyum install iperf3
.
To test network connectivity between nodes
-
Start the
iperf3
server on each node:# for i in $(vinfra node list -f json | jq -r '.[].host'); do ssh $i iperf3 -s -p 30001 -D; done
-
Start the benchmark on each node:
# NODES=`vinfra node list -f json | jq -r '.[].host' | sort | grep -v $(hostname -s)` # for i in $NODES; do iperf3 -c $i -p 30001 ; done | egrep -i 'local|sender|receiver'
The output will be similar to the following:
[ 4] local 172.24.4.1 port 40666 connected to 172.24.4.2 port 30001 [ 4] 0.00-10.00 sec 28.6 GBytes 24.5 Gbits/sec 44 sender [ 4] 0.00-10.00 sec 28.6 GBytes 24.5 Gbits/sec receiver [ 4] local 172.24.4.1 port 44968 connected to 172.24.4.4 port 30001 [ 4] 0.00-10.00 sec 22.1 GBytes 19.0 Gbits/sec 1672 sender [ 4] 0.00-10.00 sec 22.1 GBytes 19.0 Gbits/sec receiver
- Terminate the
iperf
server when done.
In the test report, check the following:
- The measured network speed between all node pairs is similar, and that it is close to the rated network speed. Otherwise, this may indicate a network problem.
- The number of retries (shown in the second-to-last column) is close to zero.
To test network throughput under load
-
Start one
iperf3
server on a different port per client on each node:# for i in $(vinfra node list -f json | jq -r '.[].host'); do ssh $i "for j in 1 2 3; \ do iperf3 -p 3000\$j -s -D ; done" ; done
-
Start the benchmark simultaneously on all of the nodes:
# PORT=30001 # use a different port on every client (ranged 30001…30003) # NODES=`vinfra node list -f json | jq -r '.[].host' | sort | grep -v $(hostname -s )` # for i in $NODES; do iperf3 -c $i -p $PORT ; done | egrep -i 'local|sender|receiver'
- Terminate the
iperf
server when done.
The test tries to maximize network load by generating traffic between all node pairs, and then reports the measured network speed between pairs.
In the test report, check the following:
- The cumulative input and output of each node is close to the rated network speed.
- The number of retries (shown in the second-to-last column) is close to zero.