3. Benchmarking Cluster Nodes

Prepare for testing as follows:

  1. Install fio 3.3 or newer on all cluster nodes and on the machine you will run the benchmarks from (unless it is one of the cluster nodes):

    # yum install fio -y
    
  2. Create a target directory for the dataset and set its redundancy mode to 3 replicas (instead of 1 replica set by default). If you have multiple tiers, also specify the one you want to test, e.g., 1.

    # mkdir /mnt/vstorage/benchmark_dir
    # vstorage -c <cluster_name> set-attr -R /mnt/vstorage/benchmark_dir replicas=3 tier=1
    

    If you want to test multiple tiers, create and use a dedicated benchmark directory for every tier.

  3. Download fio job profiles to a host or virtual environment that you will run the benchmark from. This host or VE must have network access to all cluster nodes.

    # wget -rnp -nH -R "index*" --cut-dirs=2 -e robots=off -A '*.fio' \
    https://docs.virtuozzo.com/fio/node/
    

    Note

    Cluster node job profiles cannot be used to benchmark VMs.

  4. On each cluster node run:

    # fio --server
    

    Fio will use port 8765 by default. In any case make sure that the port is open. For example, you can open it with an iptables rule:

    # iptables -I INPUT -p tcp -m tcp -m multiport --dports 8765 -j ACCEPT
    
  5. Specify the required size and numjobs values in every node job profile:

    • size is at least double node’s RAM divided by node’s CPU cores,
    • numjobs is the number of node’s CPU cores.

    If Hyper-Threading is enabled, use the number of CPU threads instead. The goal is to fully load the CPU without overcommitting it.

  6. Create the dataset. From the directory with the fio profiles, run

    # for N in {<node1>,<node2>,...,<nodeN>}; do fio --minimal --client=$N prepare-set.fio; done
    

    Where nodeN is the host name or IP address of a cluster node.

    This command will prepare identical test datasets on all cluster nodes in succession. This may take several minutes but will provide less deviation of results during actual benchmarking. Ignore any performance results displayed by this fio run.

  7. If cluster nodes have SSD/NVMe caches, make sure they have been flushed before running any benchmarks. You can check the caches as follows:

    1. On a cluster node, run

      # vstorage -c <cluster_name> top
      
    2. While in the text-based dashboard, press c to expand the chunks tab and then cycle columns with i until you see the JRN_FULL and FLAGS columns.

    3. Wait until JRN_FULL is 0% and each CS ID is marked by a c flag, e.g., “JC​c​”. This may take some time when the cluster is not under I/O load.

Now you are ready to run the benchmark. It is recommended to run read tests before write and read/write tests.

From the directory with the fio profiles, run the benchmark on all cluster nodes as follows (<nodeN> is the host name or IP address of a cluster node):

  • Sequential read test:

    # fio  --client=<node1> seqread.fio --client=<node2> seqread.fio [...]
    
  • Random read test:

    # fio  --client=<node1> randread.fio --client=<node2> randread.fio [...]
    
  • Sequential write test (run after making sure that caches have been flushed):

    # fio  --client=<node1> seqwrite.fio --client=<node2> seqwrite.fio [...]
    
  • Random write test (run after making sure that caches have been flushed):

    # fio  --client=<node1> randwrite.fio --client=<node2> randwrite.fio [...]
    
  • Mixed read/write test (run after making sure that caches have been flushed):

    # fio  --client=<node1> randrw.fio --client=<node2> randrw.fio [...]
    
  • Expand test:

    # fio  --client=<node1> expand.fio --client=<node2> expand.fio [...]