Running the benchmark for NFS and iSCSI


To run fio scripts

  1. On each load generator node, start the fio server:

    # fio --server
  2. On the coordinator node, create the dataset. In this example, we assume the scripts are stored in the /root/scripts directory, and we want to run the preparation script for the three load generators at,, and

    # for N in {,,}; do fio --minimal --client=$N /root/scripts/prepare-set.fio; done

    You can also specify the hostnames of cluster nodes, instead if their IP addresses.

    This command will prepare identical test datasets on all cluster nodes in succession. This may take several minutes, but it will provide less deviation of results during actual benchmarking. Ignore any performance results displayed by this fio run.

  3. On the coordinator node, start the desired benchmark script. In this example, we assume the scripts are stored in the /root/scripts directory, and we want to run randread.fio for the three load generators at,, and

    # fio --client --client --client /root/scripts/randread.fio

To collect fio results

Fio shows several reports, one for each load generator participating in the benchmark. At the bottom, you will find the section "All clients", which contains the summary of the overall performance.

An example output is as follows:

All clients: (groupid=0, jobs=4): err= 0: pid=5190: Mon Mar 7 15:16:09 2022
  read: IOPS=18.9k, BW=73.9MiB/s (77.5MB/s)(8865MiB/120001msec)
    slat (usec): min=105, max=97552, avg=206.52, stdev=187.96
    clat (usec): min=5, max=299572, avg=53923.77, stdev=8760.84
     lat (usec): min=210, max=299745, avg=54131.08, stdev=8784.92
    clat percentiles (msec):
     | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 48],
     | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 53], 60.00th=[ 54],
     | 70.00th=[ 56], 80.00th=[ 59], 90.00th=[ 64], 95.00th=[ 68],
     | 99.00th=[ 79], 99.50th=[ 82], 99.90th=[ 89], 99.95th=[ 106],
     | 99.99th=[ 296]
   bw ( KiB/s): min=39528, max=88968, per=99.97%, avg=75622.67, stdev=1440.79,
   iops : min= 9882, max=22242, avg=18905.56, stdev=360.20, samples=957
  lat (usec) : 10=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=32.91%
  lat (msec) : 100=67.02%, 250=0.01%, 500=0.04%
  cpu : usr=2.35%, sys=9.55%, ctx=2283821, majf=0, minf=1079
  IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=2269396,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=73.9MiB/s (77.5MB/s), 73.9MiB/s-73.9MiB/s (77.5MB/s-77.5MB/s),
io=8865MiB (9295MB),

Disk stats (read/write):
   sda: ios=2267039/1298, merge=0/967, ticks=417590/1434, in_queue=418444,

In this output, some of the most relevant metrics are highlighted in bold. They include IOPS, latency (lat), I/O depth, and bandwidth (bw).