7.6. Using SSD Drives

Virtuozzo Storage supports SSD drives formatted to the ext4 filesystem and optionally mounted with TRIM support enabled.

Note

Virtuozzo Storage SSD usage scenario does not generate TRIM commands. Also, modern drives like Intel SSD DC S3700 do not need TRIM at all.

Along with using SSD drives for storing data chunks, Virtuozzo Storage also supports the use of such drives for special purposes:

  • Write journaling. You can attach an SSD drive to a chunk server and configure the drive to store a write journal. By doing so, you can boost the performance of write operations in the cluster by up to 2 and more times.
  • Data caching. You can attach an SSD drive to a client and configure the drive to store a local cache of frequently accessed data. By having a local cache on a client’s SSD drive, you can increase the overall cluster performance by up to 10 and more times.

Solid-state drive space should be split between write journals and read cache based on the nature of workload, i.e. the proportion of write operations to read operations. The read cache size also depends on how demanding your applications are. If you are unsure, split SSD space in equal parts. For example, if you have a 100GB SSD and four chunk servers on four 1TB HDDs, divide the SSD space as follows:

  • 20 GB reserved for checksums and emergency needs and also to prevent the SSD from filling up completely (because its performance would then degrade),
  • 40 GB for read cache,
  • 40 GB for write journals, i.e. 10 GB per HDD/chunk server.

Checksums require 4 bytes of space for each 4KB page (1:1000 ratio). For example, 4 TB of storage will require 4 GB of space for checksums.

To understand how much space should be allocated for write journals and read cache in a specific cluster configuration, run the vstorage advise-configuration command. Using cluster parameters as input data, the command will output suggestions on how to optimize the cluster performance, set up the host, and mount the cluster via /etc/fstab. (see the following sections for examples).

In general, optimization for write journals means that about 70% of SSD space should be used for said journals and about 30% for read cache; Optimization for reads means the opposite proportion. If optimization for either writes or reads is not the goal, the idea is to use equal parts of SSD space for write journals and read cache. In each case, some space should be reserved for checksums and such (the amount to reserve is also suggested by the vstorage advise-configuration command).

Finally, the table below will help you understand how many SSDs you will need for your cluster.

SSD Type Number of SSDs
Intel SSD 320 Series, Intel SSD 710 Series, Kingston SSDNow E enterprise series, or other SATA 3Gbps SSD models providing 150-200MB/s of sequential write of random data. 1 SSD per up to 3 HDDs
Intel SSD DC S3700 Series, Samsung SM1625 enterprise series, or other SATA 6Gbps SSD models providing at least 300MB/sec of sequential write of random data. 1 SSD per up to 5-6 HDDs

The following sections provide detailed information on configuring SSD drives for write journaling and data caching.

Note

  1. Not all solid-state drives obey flush semantics and commit data in accordance with the protocol. This may result in arbitrary data loss or corruption in case of a power failure. Always check your SSDs with the vstorage-hwflush-check tool (for more information, see Checking Data Flushing).
  2. We recommend using Intel SSD DC S3700 drives. However, you can also use Samsung SM1625, Intel SSD 710, Kingston SSDNow E or any other SSD drive with support for data protection on power loss. Some of the names of this technology are: Enhanced Power Loss Data Protection (Intel), Cache Power Protection (Samsung), Power-Failure Support (Kingston), Complete Power Fail Protection (OCZ). For more information, see SSD Drives Ignore Disk Flushes.
  3. The minimal recommended SSD size is 30 GB.

7.6.1. Configuring SSD Drives for Write Journaling

Using SSD drives for write journaling can help you reduce write latencies, thus improving the overall cluster performance.

To determine how much SSD space you will need for CS journals, use the vstorage advise-configuration command with the -w option. For example:

# vstorage -c stor1 advise-configuration -w --cs /vstorage/stor1-cs1 --cs /vstorage/stor1-cs2 --cs /vstorage/stor1-cs3 --cs /vstorage/stor1-cs4 --ssd /vstorage/stor1-ssd -m /vstorage/stor1
You have the following setup:
CS  on /vstorage/stor1-cs4 -- Total disk space 1007.3GB
CS  on /vstorage/stor1-cs3 -- Total disk space 1007.3GB
CS  on /vstorage/stor1-cs2 -- Total disk space 1007.3GB
CS  on /vstorage/stor1-cs1 -- Total disk space 1007.3GB
SSD on /vstorage/stor1-ssd -- Total disk space 251.8GB
Proposed server configuration optimized for writes:
 -  155.9GB (61%) for CS journals, 66.8GB (26%) for mount read cache on /vstorage/stor1-ssd,
 29.1GB (11%) reserved (including  3.9GB checksums for 3.9TB of data)
 - CS journal sizes:
         38.9GB for /vstorage/stor1-cs4 at /vstorage/stor1-ssd
         38.9GB for /vstorage/stor1-cs3 at /vstorage/stor1-ssd
         38.9GB for /vstorage/stor1-cs2 at /vstorage/stor1-ssd
         38.9GB for /vstorage/stor1-cs1 at /vstorage/stor1-ssd
How to setup the node:
vstorage -c stor1 make-cs -r /vstorage/stor1-cs4/cs -j /vstorage/stor1-ssd/cs4-stor1-journal -s 39914
vstorage -c stor1 make-cs -r /vstorage/stor1-cs3/cs -j /vstorage/stor1-ssd/cs3-stor1-journal -s 39914
vstorage -c stor1 make-cs -r /vstorage/stor1-cs2/cs -j /vstorage/stor1-ssd/cs2-stor1-journal -s 39914
vstorage -c stor1 make-cs -r /vstorage/stor1-cs1/cs -j /vstorage/stor1-ssd/cs1-stor1-journal -s 39914
vstorage-mount -c stor1 /vstorage/stor1 -C /vstorage/stor1-ssd/vstorage-stor1-cache -R 68424
Mount option for automatic cluster mount from /etc/fstab:
vstorage://stor1 /vstorage/stor1 fuse.vstorage cache=/vstorage/stor1-ssd/vstorage-stor1-cache, cachesize=68424 0 0

In this example, the suggestion is to allocate 61% of SSD space for CS journals to achieve optimal cluster performance.

Note

  1. If you have multiple chunk servers on a single host, create a separate SSD journal for each CS, making sure that the SSD has enough space for all CS journals.To modify the size of existing CS journals use the vstorage configure-cs command.
  2. When deciding on a journal size without using the vstorage advise-configuration command, make sure there is 1GB of SSD space per each 1TB of HDD space for checksums.

7.6.1.1. Setting Up a Chunk Server with a Journal on SSD

To set up a chunk server that stores a journal on an SSD drive, do the following:

  1. Log in to the Node you want to act as a chunk server as root or as a user with root privileges. The Node must have at least one hard disk drive (HDD) and one solid state drive (SSD).

  2. Download and install the following RPM packages: vstorage-ctl, vstorage-libs-shared, and vstorage-chunk-server.

    These packages are available in the Virtuozzo remote repository (this repository is automatically configured for your system when you install Virtuozzo) and can be installed with this command:

    # yum install vstorage-chunk-server
    
  3. Make sure that cluster discovery is configured for the server. For details, see Configuring Cluster Discovery.

  4. Authenticate the server in the cluster, if it is not yet authenticated:

    # vstorage -c stor1 auth-node
    
  5. If required, prepare the SSD as described in Preparing Disks for Virtuozzo Storage.

  6. Create the chunk server configuration, repository, and the journal, for example:

    # vstorage -c stor1 make-cs -r /vstorage/stor1-cs -j /ssd/stor1/cs1 -s 30720
    

    This command

    • Makes the /vstorage/stor1-cs directory on your computer’s hard disk drive and configures it for storing data chunks.
    • Configures your computer as a chunk server and joins it to the stor1 Virtuozzo Storage cluster.
    • Creates the journal in the /ssd/stor1/cs1 directory on the SSD drive and allocates 30 GB of disk space to this journal.

    Note

    When choosing a directory for the journal and deciding on its size, allocate the required space for the journal and make sure there is 1GB of  SSD space per each 1TB of HDD space for checksums.

  7. Start the chunk server management service vstorage-csd and configure it to start automatically on the chunk server boot:

    # systemctl start vstorage-csd.target
    # systemctl enable vstorage-csd.target
    

7.6.1.2. Adding, Destroying, and Configuring Chunk Server Journals in Live Virtuozzo Storage Clusters

Note

To obtain CS repository paths, use the vstorage list-services -C command.

Adding Chunk Server Journals

To add a new journal to a chunk server, use the vstorage configure-cs -a -s command. For example, to add a 2048MB journal to the chunk server CS#1 and place it in a directory on an SSD drive mounted to /ssd, run:

# vstorage -c stor1 configure-cs -r /vstorage/stor1-cs1/data -a /ssd/stor1-cs1-journal -s 2048

Destroying Chunk Server Journals

To destroy a chunk server journal, use the vstorage configure-cs -d command. For example:

# vstorage -c stor1 configure-cs -r /vstorage/stor1-cs1/data -d

Moving Chunk Server Journals

To change the chunk server journal directory, do the following using the commands above:

  1. Destroy the existing journal
  2. Add a new journal with the required size at the required location.

Resizing Chunk Server Journals

To resize a chunk server journal, use the vstorage configure-cs -s command. For example, to resize a CS journal to 4096MB:

# vstorage -c stor_1 configure-cs -r /vstorage/stor_1-cs1/data -s 4096

7.6.1.3. Disabling Checksumming

Using checksumming, you can provide better reliability and integrity of all data in the cluster. When checksumming is enabled, Virtuozzo Storage generates checksums each time some data in the cluster is modified. When this data is then read, the checksum is computed once more and compared with the already existing value.

By default, data checksumming is automatically enabled for newly created chunk servers that use journaling. If necessary, you can disable this functionality using the -S option when you set up a chunk server, for example:

# vstorage -c stor1 make-cs -r /vstorage/stor1-cs -j /ssd/stor1/cs1 -s 30720 -S

7.6.1.4. Configuring Data Scrubbing

Data scrubbing is the process of checking data chunks for durability and verifying their contents for readability and correctness. By default, Virtuozzo Storage is set to examine two data chunks per minute on each chunk server in the cluster. If necessary, you can configure this number using the vstorage utility, for example:

# vstorage -c stor1 set-config mds.wd.verify_chunks=3

This command sets the number of chunks to be examined on each chunk server in the stor1 cluster to 3.

7.6.2. Configuring SSD Drives for Data Caching

Another way of improving the overall cluster performance is to create a local cache on a client’s SSD drive. Once you create the cache, all cluster data accessed two or more times will be put to that cache.

The table below lists the main features specific to a local cache:

Feature Description
Quick access time Data in the local cache can be accessed much faster (up to 10 times and more) as compared to accessing the same data stored on chunk servers in the cluster.
No network bandwidth consumption Cluster network bandwidth is not consumed because the data is accessed locally.
Special boot cache Local cache uses a special boot cache to store small amounts of data on file openings. This significantly speeds up the process of starting virtual machines and Containers.
Cache survivability Local cache is persistent and can survive a graceful system shutdown; however, it is dropped when the system crashes.
Sequential access filtering

Only randomly accessed data is cached.

Data backup applications may generate a huge amount of sequential IO. Preventing such IO from being cached helps to avoid stressing the cache.

To determine how much SSD space you will need for the cache, use the vstorage advise-configuration command with the -r option. For example:

# vstorage -c stor1 advise-configuration -r --cs /vstorage/stor1-cs1 --cs /vstorage/stor1-cs2 --cs /vstorage/stor1-cs3 --cs /vstorage/stor1-cs4 --ssd /vstorage/stor1-ssd -m /vstorage/stor1
You have the following setup:
CS  on /vstorage/stor1-cs1 -- Total disk space 1007.3GB
CS  on /vstorage/stor1-cs2 -- Total disk space 1007.3GB
CS  on /vstorage/stor1-cs3 -- Total disk space 1007.3GB
CS  on /vstorage/stor1-cs4 -- Total disk space 1007.3GB
SSD on /vstorage/stor1-ssd -- Total disk space 251.8GB
Proposed server configuration optimized for reads:
 -  66.8GB (26%) for CS journals, 155.9GB (61%) for mount read cache on /vstorage/stor1-ssd,
 29.1GB (11%) reserved (including  3.9GB checksums for 3.9TB of data)
 - CS journal sizes:
         16.7GB for /vstorage/stor1-cs4 at /vstorage/stor1-ssd
         16.7GB for /vstorage/stor1-cs3 at /vstorage/stor1-ssd
         16.7GB for /vstorage/stor1-cs2 at /vstorage/stor1-ssd
         16.7GB for /vstorage/stor1-cs1 at /vstorage/stor1-ssd
How to setup the node:
vstorage -c stor1 make-cs -r /vstorage/stor1-cs4/cs -j /vstorage/stor1-ssd/cs4-stor1-journal -s 17106
vstorage -c stor1 make-cs -r /vstorage/stor1-cs3/cs -j /vstorage/stor1-ssd/cs3-stor1-journal -s 17106
vstorage -c stor1 make-cs -r /vstorage/stor1-cs2/cs -j /vstorage/stor1-ssd/cs2-stor1-journal -s 17106
vstorage -c stor1 make-cs -r /vstorage/stor1-cs1/cs -j /vstorage/stor1-ssd/cs1-stor1-journal -s 17106
vstorage-mount -c stor1 /vstorage/stor1 -C /vstorage/stor1-ssd/vstorage-stor1-cache -R 159658
Mount option for automatic cluster mount from /etc/fstab:
vstorage://stor1 /vstorage/stor1 fuse.vstorage cache=/vstorage/stor1-ssd/vstorage-stor1-cache, cachesize=159658 0 0

In this example, the suggestion is to allocate 61% of SSD space for the cache to achieve optimal cluster performance.

7.6.2.1. Creating a Local Cache

Note

Unlike directories used in most Virtuozzo Storage configuration steps, the local cache on SSD is a file. Make sure you supply correct paths to the vstorage-mount -C command and the cache parameter in the corresponding /etc/fstab entry.

You create a local cache when mounting a Virtuozzo Storage cluster to a client. This process includes two steps:

  1. If required, preparing the SSD as described in Preparing Disks for Virtuozzo Storage.
  2. Using the vstorage-mount command to mount the cluster and create the cache.

For example, to make a 64 GB local cache for the stor1 cluster and store it in the file /mnt/ssd/vstorage-cache-for-cluster-stor1, you can execute the following command:

# vstorage-mount -c stor1 /vstorage/stor1 -C /mnt/ssd/vstorage-cache-for-cluster-stor1 -R 64000

If you do not specify the cluster size, vstorage-mount will automatically calculate it using the following formula:

SSD_free_space - 10 GB - SSD_total_space/10

So if the total size of your SSD drive is 100 GB and it has 80 GB of free space, the command will create the local cache with the size of 60 GB.

Note

  1. The local cache is not created if the resulting cache size is less than 1 GB.
  2. If you also plan to configure the SSD drive for write journaling, first create the journal to reserve disk space for it and then create a local cache. For more information, see Configuring SSD Drives for Write Journaling.

Configuring Automatic Cache Creation

You can automate the procedure of creating a local cache so that it is automatically created each time you start the client. To do this, add the information about the cache to the /etc/fstab file on the client.

For example, to (1) have an automatically created cache with the name of vstorage-cache-for-cluster-stor1 and size of 64 GB, (2) store it in the /mnt/ssd directory on the client, and (3) disable checksumming for data in the local cache, specify the following parameters in /etc/fstab and separate them by commas:

  • cache=<path>. Sets the full path to the local cache file.
  • cachesize=<size>. Specifies the size of the local cache, in megabytes.
  • cachechksum=n. Disables checksumming for your data; by default, data checksumming is enabled.

Once you set these parameters, your fstab file should look like the following:

vstorage://stor1 /vstorage/stor1 fuse.vstorage cache=/mnt/ssd/ vstorage-cache-for-cluster-stor1,cachesize=64000,cachechksum=n 0 0

For more information on options you can use to set up and configure a local cache, see the vstorage-mount man pages.

Disabling Cache Checksumming

To provide better reliability and integrity of your data, the vstorage-mount command automatically enables checksumming for the data in the local cache. If necessary, you can disable data checksumming by passing the -S option to vstorage-mount:

# vstorage-mount -c stor1 /vstorage/stor1 -C /mnt/ssd/vstorage-cache-for-cluster-stor1 -R 64000 -S

Querying Cache Information

To check whether the cache for a mounted cluster is active and view its current parameters, you can use this command:

# cat /vstorage/stor1/.vstorage.info/read_cache_info
path            : /mnt/ssd/vstorage-cache-for-cluster-stor1
main  size (Mb) : 56000
boot  size (Mb) : 8000
block size (Kb) : 64
checksum        : enabled

If the cache does not exist, the command output is empty. Otherwise, the command prints:

  • path to the cache file,
  • size of the main and boot caches,
  • block size,
  • checksum status.