Configuring NVMe performance

In the storage cluster, the performance of any disk is limited by the performance of a chunk service (CS), which is single threaded. To increase the parallelism and throughput of fast disks such as non-volatile memory express (NVMe) devices, you can enable NVMe performance for specific storage tiers. You can benefit from using this feature when running workloads with random writes, such as virtual machines. With NVMe performance enabled, one physical device will be able to run multiple chunk services. Note, however, that the amount of RAM and CPU cores reserved on a node will increase linearly with the number of CSes.

A storage tier with the disk failure domain cannot have NVMe performance enabled. If you want to use such tiers in your cluster, ensure that they have NVMe performance disabled before assigning storage disks to them.

Though it is convenient to configure NVMe performance before creating the storage cluster, you can also do this at any time afterwards. However, configuring NVMe performance in the existing cluster requires releasing and reattaching each device in the tier, for the change to take effect.

Limitations

It is recommended to enable NVMe performance only for NVMe or SSD devices. Using this feature with other devices will lead to higher resource consumption without actual performance benefits.
With NVMe performance enabled, it is recommended to configure redundancy by replication. Using erasure coding in this case will not improve the storage cluster performance.
You cannot configure NVMe performance for a storage tier with the failure domain set to "Disk".
Using multiple chunk services per physical drive is only recommended for NVMe devices above 512 GB in size.
In a single-node deployment, using multiple chunk services per physical drive is not supported.

Prerequisites

The infrastructure nodes are equipped with NVMe or SSD disks.
When increasing the number of chunk services per physical device, ensure that each CS has sufficient space allocated to it, that is, at least 100 GB.
A clear understanding of the CPU and RAM reservations, which are described in General requirements.

To enable NVMe performance for a tier

Admin panel

Go to the Settings > System settings > Storage performance screen.
In the Custom NVMe performance section, select the desired number of chunk services per physical device for the required storage tier. The maximum number is 4. The recommended number is 2 for NVMe devices and 4 for PMem devices.
Click Save to apply the changes.
If you have enabled NVMe performance after creating the storage cluster, release and reattach each NVMe disk in the affected storage tiers.

Command-line interface

Use the following command:

vinfra cluster settings number-of-cses-per-disk set [--tier0 <number>] [--tier1 <number>] 
                                                    [--tier2 <number>] [--tier3 <number>]

--tier0 <number>: Set the number of CSes per disk for tier 0.
--tier1 <number>: Set the number of CSes per disk for tier 1.
--tier2 <number>: Set the number of CSes per disk for tier 2.
--tier3 <number>: Set the number of CSes per disk for tier 3.

For example, to set 4 CSes per disk for the tier 0 and 4 CSes per disk for the tier 1, run:

# vinfra cluster settings number-of-cses-per-disk set --tier0 4 --tier1 4

You can check that the setting has been applied by running vinfra cluster settings number-of-cses-per-disk show:

# vinfra cluster settings number-of-cses-per-disk show
+---------+-------+
| Field   | Value |
+---------+-------+
| maximal | 10    |
| minimal | 1     |
| tier0   | 4     |
| tier1   | 4     |
| tier2   | 1     |
| tier3   | 1     |
+---------+-------+

If you have enabled NVMe performance after creating the storage cluster, release and reattach each NVMe disk in the affected storage tiers.

To disable NVMe performance for a tier

Admin panel

Go to the Settings > System settings > Storage performance screen.
Select 1 as the number of chunk services per physical device for the required storage tier.
Click Save to apply the changes.
If you have disabled NVMe performance after creating the storage cluster, release and reattach each NVMe disk in the affected storage tiers.

Command-line interface

Use the following command:

vinfra cluster settings number-of-cses-per-disk set [--tier0 <number>] [--tier1 <number>] 
                                                    [--tier2 <number>] [--tier3 <number>]

--tier0 <number>: Set the number of CSes per disk for tier 0.
--tier1 <number>: Set the number of CSes per disk for tier 1.
--tier2 <number>: Set the number of CSes per disk for tier 2.
--tier3 <number>: Set the number of CSes per disk for tier 3.

For example, to disable NVMe performance for the tier 1, run:

# vinfra cluster settings number-of-cses-per-disk set --tier1 1

You can check that the setting has been applied by running vinfra cluster settings number-of-cses-per-disk show:

# vinfra cluster settings number-of-cses-per-disk show
+---------+-------+
| Field   | Value |
+---------+-------+
| maximal | 10    |
| minimal | 1     |
| tier0   | 4     |
| tier1   | 1     |
| tier2   | 1     |
| tier3   | 1     |
+---------+-------+

If you have disabled NVMe performance after creating the storage cluster, release and reattach each NVMe disk in the affected storage tiers.