Checking the RDMA network
You can check the RDMA network state via vendor-provided command line tools.
To check the network hardware
Lists the RDMA devices available for use:
# ibv_devices
device node GUID
------ ----------------
rdmao1 5e6f69fffe27b644
rdmao2 5e6f69fffe27b645
To check the network connectivity
Start the RDMA-ping server on any node:
# rping -s -C 10 -v
Then, start the RDMA-ping client on any other node:
# rping -c -a <server_IP> -C 10 -v
Where <server_IP> is the IP address of the RDMA-ping server.
The tool will have an output similar to this:
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA client DISCONNECT EVENT...
To check the network bandwidth
Start the benchmark server on any node:
# ib_send_bw -d mlx4_0 -i 1 -F --report_gbits
Then, start the benchmark client on any other node:
# ib_send_bw -d mlx4_0 -i 1 -F --report_gbits <server_IP>
Where <server_IP> is the IP address of the benchmark server.
The tool will have an output similar to this:
Send BW Test Dual-port : OFF Device : mlx4_0 Number of qps : 1 Transport type : IB Connection type : RC RX depth : 512 CQ Moderation : 100 Mtu : 1024[B] Link type : Ethernet Gid index : 0 Max inline data : 0[B] rdma_cm QPs : OFF Data ex. method : Ethernet -------------------------------------------------------------------------- local address: LID 0000 QPN 0x0065 PSN 0xc8f367 GID: 254:128:00:00:00:00:00:00:246:82:20:255:254:23:27:129 remote address: LID 0000 QPN 0x005d PSN 0x884d7d GID: 254:128:00:00:00:00:00:00:246:82:20:255:254:23:31:225 -------------------------------------------------------------------------- #bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps] 65536 1000 0.00 36.40 0.069428
What's next