Checking the RDMA network

You can check the RDMA network state via vendor-provided command line tools.

To check the network hardware

Lists the RDMA devices available for use:

# ibv_devices
    device          	   node GUID
    ------          	----------------
    rdmao1          	5e6f69fffe27b644
    rdmao2          	5e6f69fffe27b645

To check the network connectivity

Start the RDMA-ping server on any node:

# rping -s -C 10 -v

Then, start the RDMA-ping client on any other node:

# rping -c -a <server_IP> -C 10 -v

Where <server_IP> is the IP address of the RDMA-ping server.

The tool will have an output similar to this:

ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst
ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu
ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuv
ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvw
ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwx
ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxy
ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz
ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyzA

To check the network bandwidth

Start the benchmark server on any node:

# ib_send_bw -d mlx4_0 -i 1 -F --report_gbits

Then, start the benchmark client on any other node:

# ib_send_bw -d mlx4_0 -i 1 -F --report_gbits <server_IP>

Where <server_IP> is the IP address of the benchmark server.

The tool will have an output similar to this:

Send BW Test
Dual-port : OFF Device : mlx4_0
Number of qps : 1 Transport type : IB
Connection type : RC
RX depth : 512
CQ Moderation : 100
Mtu : 1024[B]
Link type : Ethernet
Gid index : 0
Max inline data : 0[B]
rdma_cm QPs : OFF
Data ex. method : Ethernet
local address: LID 0000 QPN 0x0065 PSN 0xc8f367
GID: 254:128:00:00:00:00:00:00:246:82:20:255:254:23:27:129
remote address: LID 0000 QPN 0x005d PSN 0x884d7d
GID: 254:128:00:00:00:00:00:00:246:82:20:255:254:23:31:225
#bytes #iterations BW peak[Gb/sec] BW average[Gb/sec] MsgRate[Mpps]
65536  1000        0.00            36.40              0.069428