2.10. Managing Virtual Machine and Container Backups¶

2.10.1.1. Improving Backup Performance¶

A virtual environment with a write-intensive workload may write to its snapshot blocks faster than they are backed up, so writing is slowed down to the backup speed. Besides, if the backup process stalls, the virtual environment also stalls. To avoid this, the original block can be copied to a temporary intermediate snapshot called a reversed delta and then rewritten without delay.

To switch to backups with reversed delta, run

# prlsrvctl set --backup-mode push-with-reversed-delta

Temporary snapshots will be saved to the virtual environment’s home directory by default. To set a different location, use the command

# prlsrvctl set --backup-tmpdir <path>

Temporary snapshots will be written with the O_DIRECT flag to avoid wasting node’s cache, so the directory must allow O_DIRECT writes.

To revert to direct write backups, run

# prlsrvctl set --backup-mode push

Or you can back up a virtual environment with the --no-reversed-delta option. For example:

# prlctl backup MyVM --no-reversed-delta

When direct write backups are created, the amount of page cache available to them is limited to 512 MB to reduce wasting node’s cache.

2.10.6.1. Attaching Backups to Linux Virtual Machines¶

1. Make sure that the prl_backup and kpartx utilities are installed in the virtual machine the backup will be attached to. The prl_backup utility is provided by Virtuozzo guest tools.

2. Obtain the ID and file name of the backup to attach. You can do this with the prlctl backup-list command. For example:

   # prlctl backup-list vm2 -f
   ...
   Backup_ID: {0fcd6696-c9dc-4827-9fbd-6ee3abe017fa}
   ...
   Name: harddisk.hdd.qcow2
   

3. Attach the backup as an HDD to the Linux VM you will access the backup from. You can do this with the prlctl set --backup-add command. For example:

   # prlctl set vm1 --backup-add {0fcd6696-c9dc-4827-9fbd-6ee3abe017fa} \
   --disk harddisk.hdd.qcow2
   Creating hdd1 (+) sata:2 real='backup:///{0fcd6696-c9dc-4827-9fbd-6ee3abe017fa}/ \
   harddisk.hdd.qcow2' backup='{0fcd6696-c9dc-4827-9fbd-6ee3abe017fa}' \
   disk='harddisk.hdd.qcow2'
   

   If the backup contains multiple disks and you need to connect them all, omit the --disk option.

4. Obtain the name of the newly attached device, which is disabled at the moment, using the prl_backup list command. For example:

   # prlctl exec vm1 prl_backup list
   ...
   List of disabled attached backups:
   [1] /dev/sdc
   

5. Enable the backup with the prl_backup enable command. For example:

   # prlctl exec vm1 prl_backup enable /dev/sdc
   

6. Optionally, make sure the backup is now enabled, using the prl_backup list -e command. For example:

   # prlctl exec vm1 prl_backup list -e
   List of enabled attached backups:
   [1] /dev/sdc (/dev/mapper/backup1)
   NAME                TYPE  SIZE   FSTYPE   UUID
   MOUNTPOINT
   backup1 (dm-3)      dm    64G
   |-backup1p1 (dm-4)  part  500M   ext4     1ac82165-113d-40ee-8ae2-8a72f62d95bf
    -backup1p2 (dm-5)  part  63.5G  LVM2_mem Zw9QiY-BiU5-o8dn-ScTK-vOZx-KujW-wbgmS3
   

Now you can mount the required backup part as a file system.

Use mount to mount the ext4 part. For example:

# prlctl exec vm1 mount /dev/mapper/backup1p1 /mnt/backup1p1

You can now access the backup part contents at /mnt/backup1p1.

To avoid an XFS UUID collision, mount the XFS partitions with the nouuid mount option:

prlctl exec vm1 mount -o nouuid /dev/mapper/backup1p1 /mnt/backup1p1

Do the following to mount the LVM2_member part:

1. Assign the new volume group a new name so it can coexist with other volume groups. You can do this with the vgimportclone command. For example:

   # prlctl exec vm1 vgimportclone -n backup1p2 /dev/mapper/backup1p2
   ...
   Volume group "VolGroup" successfully renamed to "backup1p2"
   ...
   Found volume group "backup1p2" using metadata type lvm2
   ...
   

   Note

   If the guest operating system is RockyLinux 9.3, use an additional option --importdevices when running the vgimportclone command. For example:

   prlctl exec vm1 vgimportclone --importdevices -n rl-backup /dev/mapper/backup1p2
   

2. Obtain the list of mountable logical volumes with the lvs command. For example:

   # prlctl exec vm1 lvs | grep backup1p2
   lv_home backup1p2 -wi-------11.54g
   lv_root backup1p2 -wi------- 50.00g
   lv_swap backup1p2 -wi-------1.97g
   

3. Activate the required logical volume with the lvchange -ay command. For example:

   # prlctl exec vm1 lvchange -ay /dev/backup1p2/lv_root
   

4. Mount the logical volume as a file system. For example:

   # prlctl exec vm1 mount /dev/backup1p2/lv_root /mnt/backup1p2
   

You can now access the backup part contents at /mnt/backup1p2.

2.10.6.2. Attaching Backups to Windows Virtual Machines¶

1. Obtain the backup ID and file name. For example:

   # prlctl backup-list vm2 -f
   ...
   Backup_ID: {cff742a9-f942-41c5-9ac2-ace3b4eba783}
   ...
   Name: harddisk.hdd.qcow2
   

2. Attach the required backup as an HDD to the Windows virtual machine you will access the backup from. For example:

   # prlctl set vm1 --backup-add {cff742a9-f942-41c5-9ac2-ace3b4eba783} \
   --disk harddisk.hdd.qcow2
   Creating hdd1 (+) sata:2 real='backup:///{cff742a9-f942-41c5-9ac2-ace3b4eba783}/ \
   harddisk.hdd.qcow2' backup='{cff742a9-f942-41c5-9ac2-ace3b4eba783}' \
   disk='harddisk.hdd.qcow2'
   

The attached backup will appear as a ready-to-use disk in the Windows virtual machine.

2.10.6.3. Attaching Backups to Linux Containers¶

1. Obtain the backup ID and file name with the prlctl backup-list -f command. For example:

   # prlctl backup-list 102 -f
   ...
   Backup_ID: {d70441dd-f077-44a0-8191-27704d4d8fdb}
   ...
   Name: root.hdd.qcow2c
   ...
   

2. Attach the backup as an HDD to the Linux container you will access the backup from. You can do this with the prlctl set --backup-add command. For example:

   # prlctl set MyCT --backup-add {d70441dd-f077-44a0-8191-27704d4d8fdb} \
   --disk root.hdd.qcow2c
   Creating hdd1 (+) sata:0 real='backup:///{d70441dd-f077-44a0-8191-27704d4d8fdb}/ \
   root.hdd.qcow2c' backup='{d70441dd-f077-44a0-8191-27704d4d8fdb}' \
   disk='root.hdd.qcow2c'
   

3. Using the backup ID, identify the ploop device corresponding to the backup. For example:

   # ploop list | grep {d70441dd-f077-44a0-8191-27704d4d8fdb}
   ploop28261 /buse/{8417a267-0919-4c8f-a31d-68671358d6a8}_ \
   {d70441dd-f077-44a0-8191-27704d4d8fdb}_root.hdd.qcow2c/content
   

4. Mount the logical volume as a file system. For example:

   # prlctl exec MyCT mount /dev/ploop28261p1 /mnt/backup1
   

You can now access the backup contents at /mnt/backup1.

2.10.8.1. Performing Backups from the Page Cache Angle¶

Backup software by its nature must, first, read all files from a disk or directory to back up and, second, write that data somewhere. If no precautions are taken, both steps generate huge amounts of page cache, as both file reads and writes send their data to the page cache.

As mentioned before, because RAM size is normally much smaller than disk size, backup forces the memory management subsystem to drop the existing page cache used by processes and replace it with the page cache generated by backup. At that, files read and written during backup are unlikely be read by other processes anytime soon. As a result, node’s RAM fills up with useless page cache, while the useful page cache is dropped.

Thus when writing backup software, one needs to make sure to keep the useful page cache on the node. That is, drop page cache generated by backup. This can be done in two ways:

1. Have backup software read and write files in the O_DIRECT mode.

   Always consider this when creating new backup tools. If you are writing a simple script, this may sometimes be possible as well.

   • If you need to copy a large file, e.g., a snapshot, use dd with the O_DIRECT flag instead of cp:

     # dd if=/path/to/src_file of=/path/to/dst_file bs=8M iflag=nocache oflag=direct
     

   • If you need to create an archive and write it somewhere, most often you can avoid page cache generation for the write part if the archiver can write to stdout. Just write the archive to disk with dd:

     # archive_and_write_to_stdout | dd of=/path/to/dst_file bs=8M oflag=direct
     

   It may not be that easy with the read part, however. For example, tar found in the mainstream tree still does not support the --o_direct option, although patches exist that implement this functionality. It is like that with rsync too. There are patches, pull requests, and even custom builds that add fadvise (FADV_DONTNEED) support to it, but none of those have been accepted to the rsync mainstream tree.

   One interesting tool is nocache. It tries to minimize the effect of applications on the Linux file system cache. It intercepts open and close system calls and calls posix_fadvise with the POSIX_FADV_DONTNEED parameter. For example:

   # nocache tar cz -O /path/to/backup_dir | dd of=/path/to/dst_file bs=8M oflag=direct
   

   or

   # nocache rsync -aiv --rsync-path='nocache rsync' /src/ host:/dest/
   

   The --rsync-path option makes the remote side use a nocache wrapper as well.

   The nocache tool provides a workaround for small and medium-sized files (note that by default it only handles files up to 1MB in size). However by the time tar, rsync, or another utility closes a large source file and nocache calls fadvise() to drop it, it is too late, as the file contents have already reached node’s page cache, replacing useful data in it. More details are provided in this post.

2. Run backup software in a cgroup with a limited page cache (Virtuozzo Hybrid Server 7).

   If your backup tools do not support the O_DIRECT file access mode, you can use a feature of Virtuozzo Hybrid Server 7: limiting of page cache for particular memory cgroups (see Limiting Memory Pressure from Node Processes).

2.10.8.2. Limiting Memory Pressure from Node Processes¶

To limit memory pressure on the hardware node resulting from fast page cache generation by node processes (like backups), one needs to do the following:

1. Create a separate memory cgroup.

2. Place the desired process into the cgroup.

3. Disable tcache for the cgroup.

4. Limit page cache for the cgroup (a feature of Virtuozzo Hybrid Server 7).

5. After the process that generates page cache fast ends, remove the memory cgroup.

The easiest way is to use the script nocache.sh that performs all these steps.

Usage:

# nocache.sh
[-limit LIMIT]
[-pcgroup PARENT_CGROUP]
<command>

Where:

• -limit LIMIT is the page cache limit in megabytes to apply to the process. The default is 256MB.

• -pcgroup PARENT_CGROUP is the parent memory cgroup, inside which the cgroup with the limited page cache will be created. The current memory cgroup is used by default.

• <command> is the command with options (or script) that generates page cache.

Examples:

# nocache.sh backup_script.sh

# nocache.sh -limit 128 -pcgroup /machine.slice backup_script.sh

If you cannot use the script for some reason, you can perform the same steps manually:

1. Create a separate memory cgroup.

   # mkdir /sys/fs/cgroup/memory/user.slice/limited_pagecache
   

   Note

   You can replace limited_pagecache with any unique string.

2. Place the desired process with PID $PID into the cgroup.

   # echo $PID >> /sys/fs/cgroup/memory/user.slice/limited_pagecache/tasks
   

   Note

   You can write a Bash wrapper for the command that creates the page-cache-heavy process. Place the process PID into the cgroup as follows and the children processes will inherit the parent’s cgroup:

   # echo $$ >> /sys/fs/cgroup/memory/user.slice/limited_pagecache/tasks
   

3. Disable tcache for the cgroup.

   # echo 1 > /sys/fs/cgroup/memory/user.slice/limited_pagecache/memory.disable_cleancache
   

4. Limit page cache for the cgroup to $LIMIT bytes (a feature of Virtuozzo Hybrid Server 7):

   # echo $LIMIT > /sys/fs/cgroup/memory/user.slice/limited_pagecache/memory.cache.limit_in_bytes
   

   Note

   The limit depends on the application. For example, 256MB is enough for tar + gzip.

5. After the process that generates page cache fast ends, remove the memory cgroup.

   # rmdir /sys/fs/cgroup/memory/user.slice/limited_pagecache
   

   Note that a cgroup with processes cannot be removed. If on step 2 you put the process PID into the cgroup, first remove it from the cgroup with:

   # echo $$ >> /sys/fs/cgroup/memory/user.slice/tasks
   

Frequenty asked questions:

• Q: May my application fail to work due to memory allocation issues if run with nocache.sh?

  A: No. The wrapper does not limit the size of anonymous memory allowed for the application, so memory allocation will not fail.

• Q: May nocache.sh affect the performance of my application?

  A: It depends on the application. Backup-like programs that read each file only once will not slow down. Applications that read same files two or more times may slow down.