Running the benchmark for NFS and iSCSI

Prerequisites

The benchmarks for the NFS or iSCSI services are set up, as described in Setting up the benchmark for NFS and iSCSI.
If the cluster nodes have SSD/NVMe caches, make sure they have been flushed, as instructed in Preparing to run the benchmark.

To run fio scripts

On each load generator node, start the fio server:
```
# fio --server
```
On the coordinator node, create the dataset. In this example, we assume the scripts are stored in the /root/scripts directory, and we want to run the preparation script for the three load generators at 10.10.10.11, 10.10.10.12, and 10.10.10.13:
```
# for N in {10.10.10.11,10.10.10.12,10.10.10.13}; do fio --minimal --client=$N /root/scripts/prepare-set.fio; done
```
You can also specify the hostnames of cluster nodes, instead if their IP addresses.

This command will prepare identical test datasets on all cluster nodes in succession. This may take several minutes, but it will provide less deviation of results during actual benchmarking. Ignore any performance results displayed by this fio run.
On the coordinator node, start the desired benchmark script. In this example, we assume the scripts are stored in the /root/scripts directory, and we want to run randread.fio for the three load generators at 10.10.10.11, 10.10.10.12, and 10.10.10.13:
```
# fio --client 10.10.10.11 --client 10.10.10.12 --client 10.10.10.13 /root/scripts/randread.fio
```

To collect fio results

Fio shows several reports, one for each load generator participating in the benchmark. At the bottom, you will find the section "All clients", which contains the summary of the overall performance.

An example output is as follows:

[…]
All clients: (groupid=0, jobs=4): err= 0: pid=5190: Mon Mar 7 15:16:09 2022
  read: IOPS=18.9k, BW=73.9MiB/s (77.5MB/s)(8865MiB/120001msec)
    slat (usec): min=105, max=97552, avg=206.52, stdev=187.96
    clat (usec): min=5, max=299572, avg=53923.77, stdev=8760.84
     lat (usec): min=210, max=299745, avg=54131.08, stdev=8784.92
    clat percentiles (msec):
     | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 48],
     | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 53], 60.00th=[ 54],
     | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 68],
     | 99.00th=[ 79], 99.50th=[ 82], 99.90th=[ 89], 99.95th=[ 106],
     | 99.99th=[ 296]
   bw ( KiB/s): min=39528, max=88968, per=99.97%, avg=75622.67, stdev=1440.79,
samples=957
   iops : min= 9882, max=22242, avg=18905.56, stdev=360.20, samples=957
  lat (usec) : 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=32.91%
  lat (msec) : 100=67.02%, 250=0.01%, 500=0.04%
  cpu : usr=2.35%, sys=9.55%, ctx=2283821, majf=0, minf=1079
  IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=2269396,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=73.9MiB/s (77.5MB/s), 73.9MiB/s-73.9MiB/s (77.5MB/s-77.5MB/s),
io=8865MiB (9295MB),
run=120001-120001msec

Disk stats (read/write):
   sda: ios=2267039/1298, merge=0/967, ticks=417590/1434, in_queue=418444,
util=99.22%

In this output, some of the most relevant metrics are highlighted in bold. They include IOPS, latency (lat), I/O depth, and bandwidth (bw).