10.1. Configuring Automatic Memory Management Policies

Virtuozzo offers several techniques for automatically managing VM and container memory. The management is performed by the vcmmd daemon by means of policies described below. Setting the correct policy may provide performance and density benefits to your Virtuozzo installation.

The following policies are currently available:

  • performance (default): best for hardware nodes with lots of small virtual environments (relative to a NUMA node size).

    Important

    It is strongly recommended to avoid overcommitting resources when the performance policy is enabled. For resource overcommit, switch to the density policy.

  • density: best for hardware nodes with memory overcommit.

  • NoOpPolicy: this policy is set automatically if no license is installed on a hardware node or if the current license is expired.

If you activated a license after installing Virtuozzo or updated an expired license, you need to restart the vcmmd daemon to apply changes. To do this:

  1. Stop all virtual environments on the host or temporarily migrate them live to another host.

  2. Run the following command:

    # systemctl restart vcmmd
    
  3. Start the virtual environments or migrate them back to the server, depending on what you did in step 1.

After restarting vcmmd, the default performance policy will be applied.

You can switch between the policies with the prlsrvctl tool.

Note

Before setting a VCMMD policy, stop all virtual machines and containers on the host or temporarily migrate them live to another host.

For example, to switch to the density policy, do the following:

  1. Make sure that there are no running virtual machines or containers on the host:

    # prlctl list
    UUID                                   STATUS   IP_ADDR     T  NAME
    
  2. Set the policy with prlsrvctl set --vcmmd-policy:

    # prlsrvctl set --vcmmd-policy density
    

To check the currently enabled policy, run

# prlsrvctl info | grep "policy"
Vcmmd policy: density config=density

10.1.1. Optimizing Virtual Machine Memory with Kernel Same-Page Merging

To optimize memory usage by virtual machines, Virtuozzo uses a feature of Linux called Kernel Same-Page Merging (KSM). The KSM daemon ksmd periodically scans memory for pages with identical content and merges those into a single page. Said page is marked as copy-on-write (COW), so when its contents are changed by a virtual machine, the kernel creates a new copy for that virtual machine.

KSM enables the host to:

  • avoid swapping due to merging of identical pages
  • run more virtual machines
  • overcommit virtual machine memory
  • speed up RAM and hence certain applications and guest operating systems

In Virtuozzo 7, KSM is managed by vcmmd and works with both performance and density policies. However, with performance policy enabled, KSM merges identical pages for each NUMA node independently which slows its operation down by as many times as the number of NUMA nodes on the host (e.g., by four times if there are four NUMA nodes).

10.1.2. Managing Host Services with VCMMD

In Virtuozzo 7, you can set memory limits and guarantees managed by vcmmd to host services. The daemon will take into account the service memory while allocating resources to virtual environments.

Note

The setting is removed on hardware node reboot.

For example, to have a service named service managed by vcmmd, do as follows:

  1. Place the service processes in /sys/fs/cgroup/memory/service.slice directory.

  2. Register service.slice in vcmmd, specifying the desired guarantee and limit in bytes. For example:

    # vcmmdctl register SRVC service.slice --guarantee 100000000 --limit 100000000 --swap 100000
    
  3. Activate the service:

    # vcmmdctl activate service.slice
    

To check if the service is managed with vcmmd, run

# vcmmdctl list
name                                     type  active  guarantee     limit    swap
<...>
service.slice                            SRVC     yes      97656     97656      97

You can change the service memory parameters at runtime using the vcmmdctl update command.

For example, to change the service.slice parameters, run:

# vcmmdctl update service.slice --guarantee 200000000 --limit 200000000

To check if the parameters have been correctly applied, run

# vcmmdctl list
name                                     type  active  guarantee     limit    swap
<...>
service.slice                            SRVC     yes     195312    195312     max

You can unregister the service from vcmmd at runtime with the vcmmdctl unregister command.

You can also set permanent limits and guarantees for services residing in system and user memory slices. Doing so will ensure that you can still access the node via SSH if it becomes heavily overcommitted. Do as follows:

  1. Specify the minimum and maximum values in bytes in the /etc/vz/vstorage-limits.conf file (-1 stands for unlimited memory). For example:

    {
      "VStorage": {
        <...>
      },
      "System": {
        "Path": "system.slice",
        "Limit": {
          "Max": -1,
          "Min": 0,
          "Share": 0.7
        },
        "Guarantee": {
          "Max": 1073741824,
          "Min": 536870912,
          "Share": 0.25
        },
        "Swap": {
          "Max": 0,
          "Min": 0,
          "Share": 0
        }
      },
      "User": {
        "Path": "user.slice",
        "Limit": {
          "Max": -1,
          "Min": 0,
          "Share": 0.7
        },
        "Guarantee": {
          "Max": 536870912,
          "Min": 268435456,
          "Share": 0.25
        },
        "Swap": {
          "Max": 0,
          "Min": 0,
          "Share": 0
        }
      }
    }
    
  2. Restart vcmmd to apply changes:

    # systemctl restart vcmmd
    
  3. Check that the parameters were applied. The actual guarantees and limits are calculated based on the minimum and maximum values specified in /etc/vz/vstorage-limits.conf.

    # vcmmdctl list
    name                                   type active guarantee     limit      swap
    system.slice                           SRVC    yes   1048576   5605796         0
    user.slice                             SRVC    yes    524288   5605796         0
    <...>
    

    Note

    This command outputs values in kibibytes.

10.1.3. Managing Virtuozzo Storage Services with VCMMD

For Virtuozzo Storage hardware nodes, vstorage.slice/vstorage-services.slice is automatically created in the memory cgroup on hardware node start.

If you deployed Virtuozzo Storage on top of a Virtuozzo server and started the vstorage services manually without rebooting the server, you need to restart the vcmmd daemon to apply changes. To do this:

  1. Stop all virtual environments on the host or temporarily migrate them live to another host.

  2. Run the following command:

    # systemctl restart vcmmd
    
  3. Start the virtual environments or migrate them back to the server, depending on what you did in step 1.

After restarting vcmmd, vstorage.slice/vstorage-services.slice will be created in the memory cgroup.

To check if vstorage is managed with vcmmd, run

# vcmmdctl list
name                                     type  active  guarantee     limit    swap
<...>
vstorage.slice/vstorage-services.slice   SRVC     yes    4194304 103854973       0

You can temporarily change memory limits and guarantees for vstorage services at runtime using the vcmmdctl update command. For example:

# vcmmdctl update vstorage.slice/vstorage-services.slice --guarantee 100000000 --limit 100000000

The parameters will be reset to the default values on the next vcmmd restart or node reboot.

To configure limits and guarantees for vstorage services permanently, do as follows:

  1. Specify the values in bytes in the /etc/vz/vstorage-limits.conf file (-1 stands for unlimited memory). For example:

    {
      "VStorage": {
        "Path": "vstorage.slice/vstorage-services.slice",
        "Limit": {
          "Max": -1,
          "Min": 0,
          "Share": 0.7
        },
        "Guarantee": {
          "Max": 1073741824,
          "Min": 536870912,
          "Share": 0.25
        },
        "Swap": {
          "Max": 0,
          "Min": 0,
          "Share": 0
        }
      },
      "System": {
        <...>
      },
      "User": {
        <...>
      }
    }
    
  2. Restart vcmmd to apply changes:

    # systemctl restart vcmmd
    
  3. Check that the parameters were applied. The actual guarantees and limits are calculated based on the minimum and maximum values specified in /etc/vz/vstorage-limits.conf.

    # vcmmdctl list
    name                                   type active guarantee     limit      swap
    <...>
    vstorage.slice/vstorage-services.slice   SRVC    yes   1048576   5605796         0
    

    Note

    This command outputs values in kibibytes.

If you use large CS journals, you may also attempt to improve storage write performance by tweaking VCMMD configuration.

Initially, storage services can use host’s entire memory. As soon as you create a virtual environment, the memory available to storage services becomes limited to 512MB, which may throttle their performance. However, you can allow storage to use more of host’s RAM by means of the StorageCacheLimitTotal parameter. It is generally recommended to set it to at least 4 GB per CS with a journal.

For example, if you have 10 CSes with journals, set the parameter to 40 GiB by adding it to the LoadManager section of /etc/vz/vcmmd.conf:

"LoadManager": {
     "Policy": "performance"
     "StorageCacheLimitTotal": 42949672960,
}

Restart vcmmd to apply changes:

# systemctl restart vcmmd

Check that the changes were applied:

# vcmmdctl config -j | grep StorageCacheLimitTotal
        "StorageCacheLimitTotal": 42949672960,