5. Understanding results

Fio will start displaying results after you execute the benchmarking command. The more nodes participate in the test, the more results will be output by fio. However, a summary will be provided at the end indicated by the words “All clients”. For example:

All clients: (groupid=0, jobs=80): err= 0: pid=0: Mon Jan 28 19:18:23 2019
 read: IOPS=198k, BW=772Mi (809M)(45.3GiB/60078msec)
  slat (usec): min=2, max=1510, avg= 6.77, stdev= 3.29
  clat (nsec): min=1567, max=287148k, avg=3787988.85, stdev=6333559.68
   lat (usec): min=50, max=287152, avg=3794.83, stdev=6333.46
 bw (  KiB/s): min= 2048, max=113536, per=0.08%, avg=9880.44, stdev=4052.50, samples=9600
 iops        : min=  512, max=28384, avg=2470.06, stdev=1013.12, samples=9600
write: IOPS=84.8k, BW=331Mi (347M)(19.4GiB/60078msec)
  slat (usec): min=2, max=1055, avg= 7.21, stdev= 3.33
  clat (usec): min=138, max=498280, avg=21347.45, stdev=40779.33
   lat (usec): min=144, max=498296, avg=21354.72, stdev=40779.01
 bw (  KiB/s): min=  968, max=49824, per=0.08%, avg=4240.17, stdev=1738.29, samples=9600
 iops        : min=  242, max=12456, avg=1059.99, stdev=434.58, samples=9600
lat (usec)   : 2=0.01%, 20=0.01%, 50=0.01%, 100=0.71%, 250=9.39%
lat (usec)   : 500=7.81%, 750=3.96%, 1000=3.80%
lat (msec)   : 2=12.40%, 4=20.13%, 10=24.61%, 20=9.37%, 50=4.52%
lat (msec)   : 100=1.43%, 250=1.70%, 500=0.17%
cpu          : usr=0.68%, sys=3.44%, ctx=15563500, majf=0, minf=491
IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
   submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
   complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
   issued rwt: total=11866050,5093585,0, short=0,0,0, dropped=0,0,0

This is a sample output from a randrw.fio (random read/write) job run on an all-flash cluster of five nodes. The main metrics are IOPS (iops) and bandwidth (bw):

read: IOPS=198k, BW=772Mi (809M)(45.3GiB/60078msec)
write: IOPS=84.8k, BW=331Mi (347M)(19.4GiB/60078msec)

Pay attention to IOPS when you benchmark random workloads and observe bandwidth when you test sequential operations. When you test random reads/writes also note the iodepth parameter that indicates how many requests are queued and waiting.

In addition:

  • slat is submission latency that indicates how much time it took to submit I/O to the kernel.

  • clat is completion latency that indicates how much time passed between submission to the kernel and completing the I/O (excluding submission latency).

  • lat is the sum of slat and clat.