3.4. Managing Network Accounting and Bandwidth

This section explains how to perform the following tasks in Virtuozzo Hybrid Server:

  • Configure network classes

  • View network traffic statistics

  • Toggling network bandwidth management

  • Configure bandwidth limits

3.4.1. Network Traffic Parameters

The table below summarizes the network traffic parameters that you can control in Virtuozzo Hybrid Server.

Parameter

Description

traffic_shaping

If set to yes, traffic limitations for outgoing traffic are set for virtual machines and containers. The default is no.

bandwidth

This parameter lists all network adapters installed on the hardware node and their bandwidth.

totalrate

This parameter defines the bandwidth to allocate for each network class. It is active if traffic shaping is turned on.

rate

If traffic shaping is turned on, this parameter specifies the bandwidth guarantee for virtual machines and containers.

ratebound

If this parameter is set to yes, the bandwidth guarantee (the global rate parameter) is also the limit for the virtual machine or container, and the virtual machine or container cannot borrow the bandwidth from the totalrate bandwidth pool.

3.4.2. Configuring Network Classes

Virtuozzo Hybrid Server allows you to track the inbound and outbound network traffic and shape the outgoing traffic for virtual machines and containers. To provide the ability to distinguish between types of traffic, e.g., domestic and international, a concept of network classes is introduced. IP address ranges fall into several network classes. Each network class represents traffic to specific resources. Access to such resources is charged at a different rate than the rest of the traffic.

Classes are specified in the /etc/vz/conf/networks_classes file. The file is in the ASCII format, and all empty lines and lines starting with the # sign are ignored. Other lines have the following format:

<class_id> <IP_address>/<prefix_length>

where <class_id> defines the network class ID, and the <IP_address>/<prefix_length> pair defines the range of IP addresses for this class. There may be several lines for each class.

Classes 0 and 1 have special meanings:

  • Class 0 defines the IP address range for which no accounting is performed. Usually, it corresponds to the hardware node subnet (the node itself and its virtual machines and containers). Setting up class 0 is not required; however, its correct setup improves performance.

  • Class 1 is defined by Virtuozzo Hybrid Server to match any IP address. It must be always present in the network classes definition file. Therefore, it is suggested not to change the default line in the networks_classes file.

    1 0.0.0.0/0
    

    If your virtual machines and containers are using IPv6 addresses, you can also add the following line to this file:

    1 ::/0
    

Other classes should be defined after class 1. They represent exceptions from the “matching-everything” rule of class 1. The example below illustrates a possible configuration of the network classes definition file containing rules for both IPv4 and IPv6 addresses:

# Hardware node networks
0 192.168.0.0/16
0 fe80::/64
# any IP address (all traffic)
1 0.0.0.0/0
1 ::/0
# class 2 - addresses for the "foreign" traffic
2 10.0.0.0/8
2 2001:db88::/64
# inside "foreign" network there
# is a hole belonging to "local" traffic
1 10.10.16.0/24
1 2001:db88:3333::/64

In this example, IPv4 addresses in the range of 192.168.0.0 to 192.168.255.255 and IPv6 addresses in the range of fe80:: to fe80::ffff:ffff:ffff:ffff are treated as class 0 addresses and no accounting is done for the traffic from virtual machines and containers destined to these addresses.

Class 2 matches the following IP addresses:

  • IPv4 addresses from 10.0.0.0 to 10.255.255.255 with the exception of addresses in the sub-range of 10.10.16.0 to 10.10.16.255, which are treated as class 1.

  • IPv6 addresses from 2001:db88:: to 2001:db88::ffff:ffff:ffff:ffff with the exception of addresses in the sub-range of 2001:db88:3333:: to 2001:db88:3333::ffff:ffff:ffff:ffff, which are also treated as class 1.

All other IP addresses (both IPv4 and IPv6) belong to class 1.

To apply changes after editing the /etc/vz/conf/networks_classes file, restart either the virtual machine(s) or/and container(s) for which changes have been made or the hardware node itself if the changes are global.

3.4.3. Viewing Network Traffic Statistics

In Virtuozzo Hybrid Server, you can view the current network traffic statistics for virtual machines and containers using the vznetstat utility. For example:

# vznetstat
UUID         Net.Class  Input(bytes)  Input(pkts)  Output(bytes)  Output(pkts)
0            0          566093064     2800575      3120481        41736
47406484...  0          67489         155          8033           110
fbb30afa-... 0          9369          78           12692          71

By default, vznetstat shows network statistics for both virtual machines and containers. Keep in mind that the vznetstat utility displays statistics only about virtual machines and containers that were started at least once.

The vznetstat utility displays the following information:

Column

Description

UUID

UUID assigned to virtual machine or container.

Net.Class

ID of the network class for which network statistics is calculated.

Input(bytes)

Amount of incoming traffic, in bytes.

Input(pkts)

Amount of incoming traffic, in packets.

Output(bytes)

Amount of outgoing traffic, in bytes.

Output(pkts)

Amount of outgoing traffic, in packets.

For example, from the command output above, you can see that around 9 MB of data were uploaded to the container MyCT, (2) about 12 MB were downloaded from it, and all the traffic was exchanged with servers from class 0 networks.

If necessary, you can view network traffic statistics separately for virtual machine or container by passing the -t option to vznetstat:

  • For containers only:

    # vznetstat -t ct
    CTID         Net.Class  Input(bytes)  Input(pkts)  Output(bytes)  Output(pkts)
    0            0          566093064     2800575      3120481        41736
    fbb30afa-... 0          9369          78           12692          71
    
  • For virtual machines only:

    # vznetstat -t vm
    UUID         Net.Class  Input(bytes)  Input(pkts)  Output(bytes)  Output(pkts)
    0            0          566093064     2800575      3120481        41736
    47406484...  0          67489         155          8033           110
    

You can also view network statistics for a particular virtual machine or container by specifying its ID after the -v option, for example:

# vznetstat -v fbb30afa-e770-4081-9d9e-6b9c262eb091
UUID         Net.Class  Input(bytes)  Input(pkts)  Output(bytes)  Output(pkts)
fbb30afa-... 0          9369          78           12692          71

This command displays statistics only for the container MyCT.

3.4.4. Configuring Traffic Shaping

Traffic shaping (also known as network bandwidth management) allows you to control what network bandwidth a virtual machine or container may use for outgoing traffic. This feature is disabled by default.

Note the following:

  • Traffic within a host cannot be shaped in the current version of Virtuozzo Hybrid Server. This includes traffic between virtual machines and containers on the same host and between those and the host itself.

  • Incoming traffic cannot be shaped for virtual machines and containers in the current version of Virtuozzo Hybrid Server.

  • VM and container traffic is shaped by marking packets’ fwmark field. If the hardware node uses custom packet filtering rules set with the MARK and CONNMARK iptables modules, the traffic shaping feature of Virtuozzo Hybrid Server will not work. Traffic statistics reported by vznetstat, however, will be correct even if the node uses custom traffic marks.

The following parameters control traffic shaping in Virtuozzo Hybrid Server:

  • TRAFFIC_SHAPING, enables and disables traffic shaping.

  • BANDWIDTH, sets bandwidth for specific network adapters.

  • TOTALRATE, sets the size of a bandwidth pool divided between virtual machines and containers on the host.

  • RATEMPU, limits packet rate in addition to byte rate.

  • RATE, sets a bandwidth guarantee for virtual machines and containers.

  • RATEBOUND, forces RATE as a limit.

Traffic shaping in Virtuozzo Hybrid Server works as follows. The bandwidth pool for a given network class (set by TOTALRATE) is divided among the virtual machines and containers transmitting data proportionally to their RATE settings. If the sum of RATE values of all virtual machines and containers transmitting data does not exceed TOTALRATE, each virtual machine or container gets the bandwidth equal to or greater than its RATE value (unless RATEBOUND is enabled for the VM or container). If the sum of RATE values of all virtual machines and containers transmitting data exceeds the TOTALRATE value, each virtual machine or container may get less than its RATE value.

To enable and configure traffic shaping, do the following:

  1. Set the value of TRAFFIC_SHAPING to yes in the global configuration file /etc/vz/vz.conf.

  2. Set the parameters BANDWIDTH, TOTALRATE in /etc/vz/vz.conf.

  3. If required, set the optional parameters RATEMPU, RATE, RATEBOUND in /etc/vz/vz.conf.

  4. If required, set RATE and RATEBOUND for specific virtual machines and containers with prlctl set --rate and prlctl set --ratebound commands.

  5. To apply changes, restart either the virtual machines and containers for which changes have been made or the hardware node itself if the changes are global.

The following sections provide more details on and explain how to set traffic shaping parameters listed above.

3.4.4.1. Setting the BANDWIDTH Parameter

The BANDWIDTH parameter is used for shaping traffic of specific network adapters. For example, for two Fast Ethernet cards, a typical setting may look like enp0s5 enp0s6:100000 where enp0s5 and enp0s6 are network adapter names. By default, the parameter is set to 100000 which corresponds to a 100 Mbps Fast Ethernet card.

3.4.4.2. Setting the TOTALRATE Parameter

The TOTALRATE parameter specifies the size of a bandwidth pool for specific network classes on the host. Virtual machines and containers can borrow bandwidth from the pool for communicating with hosts from the corresponding network class. The parameter thus limits the total available outgoing traffic for a network class that virtual machines and containers can consume.

The parameter is set as <NIC>:<network_class>:<bandwidth_in_Kbps>. For example, to set the pool size to 4 Mbps for network class 1 on the Ethernet adapter enp0s5, set TOTALRATE to enp0s5:1:4000. Multiple entries can be separated by spaces, e.g., enp0s5:1:4000 enp0s6:2:8000.

3.4.4.3. Setting the RATEMPU Parameter

The optional RATEMPU parameter (where “MPU” stands for “minimum packet unit”) limits the packet rate by making packets smaller than MPU in size consume HTB tokens. With it, small packets can be accounted as larger ones and limited by TOTALRATE and RATE parameters. Approximately, the maximum packets per second rate can be calculated as TOTALRATE / RATEMPU.

This parameter has the following syntax: <NIC>:<network_class>[:<MPU_in_bytes_per_packet>]. If the part <MPU_in_bytes_per_packet> is omitted, the default value of 1000 bytes is used. Multiple entries can be separated by spaces, e.g., enp0s5:1:2000 enp0s6:2:4000. To set the RATEMPU parameter for all known Ethernet devices set <NIC> to an asterisk (*). For example, to set the minimal packet size to 2 Kb for network class 1 on all the Ethernet adapters on the node, change the value to *:1:2000.

3.4.4.4. Setting the RATE and RATEBOUND Parameters

The optional RATE parameter allows you to guarantee virtual machines and containers outgoing bandwidth to destinations in a specific network class on a specific Ethernet device. The guaranteed bandwidth is not a limit (unless the RATEBOUND parameter is also set to on, see below). A virtual machine or container can additionally obtain unused bandwidth from the bandwidth pool defined by TOTALRATE.

You can set the guaranteed bandwidth in two ways:

  1. For all virtual machines and containers on the host by setting RATE in the global configuration file /etc/vz/vz.conf.

    The parameter is set as <NIC>:<network_class>:<bandwidth_in_Kbps>. For example, to guarantee all virtual machines and containers on the host the bandwidth of at least 8 Kbps for outgoing traffic in network class 1 on the Ethernet device enp0s5, set the RATE parameter to enp0s5:1:8.

  2. For specific virtual machines or containers by means of the prlctl set --rate command.

    For example, to guarantee the container MyCT the bandwidth of at least 16 Kbps for outgoing traffic in network class 1, run

    # prlctl set MyCT --rate 1:16
    

    This command sets the bandwidth for the default network adapter only. If you need to set bandwidth for other network adapters, set RATE in /etc/vz/vz.conf.

    Note

    It is recommended to increase RATE value in 8 Kbps increments and set it to at least 8 Kbps.

The optional RATEBOUND parameter specifies whether the network bandwidth guaranteed by RATE is also a limit. By default, this feature is disabled for all newly created virtual machines and containers so they may additionally obtain unused bandwidth from the pool set by TOTALRATE.

You can limit bandwidth of virtual machines and containers to the guaranteed value as follows:

  1. For all virtual machines and containers on the host by setting RATEBOUND in the global configuration file /etc/vz/vz.conf (omitted by default).

  2. For specific virtual machines or containers by means of the prlctl set --ratebound command. For example:

    # prlctl set MyCT --ratebound yes
    

If set, values of RATE and RATEBOUND provided for specific virtual machines and containers are chosen over global values in /etc/vz/vz.conf.

3.4.4.5. Traffic Shaping Example

The example below illustrates a scenario when the containers MyCT1 and MyCT2 have RATEBOUND set to no, and the virtual machine MyVM has RATEBOUND set to yes. With the default TOTALRATE of 4096 Kbps and RATE of 8 Kbps, the bandwidth pool will be distributed as follows:

MyCT1

MyCT2

MyVM

Consumed Bandwidth

transmits

idle

idle

MyCT1: 4096 Kbps

idle

idle

transmits

MyVM: 8 Kbps

transmits

transmits

idle

MyCT1: 2048 Kbps MyCT2: 2048 Kbps

transmits

idle

transmits

MyCT1: 4032 Kbps MyVM: 8 Kbps

transmits

transmits

transmits

MyCT1: 2016 Kbps MyCT2: 2016 Kbps MyVM: 8 Kbps