7.4. Updating Automatically with Smart Updates

Smart updates build on automatic updates, improving their safety and providing the user with more control. Enabling smart updates means joining Virtuozzo Hybrid Server nodes to a Kubernetes cluster and creating an update manager entity to update the nodes according to set rules.

As it is recommended to set up a highly available Kubernetes cluster in production, three or more control plane (master) nodes in the same subnet are required to set up smart updates. In addition, you will need a free IP address at which the control plane will be available.

Note the following:

  • Smart updates only work on ReadyKernel and Virtuozzo Linux.

  • Virtuozzo Hybrid Server packages are not managed by smart updates. They are still managed by automatic updates described in the previous section.

  • Node update policies are initially obtained from the Virtuozzo update server as is with automatic updates. Then, however, a local update schedule, maintenance window, and policies are used for each node. They can be configured by changing the local update manager specification. Nodes with the auto policy are assigned one of the following policies:

    • fast, updates installed immediately after publication, assigned to 10% of nodes

    • slow, updates installed two weeks after publication, assigned to 40% of nodes

    • stable, updates installed six weeks after publication, assigned to 50% of nodes

  • Updates are first checked for during the setup and then each 24 hours. Also, a check is performed each time the list of excluded RPM packages or ReadyKernel patches is modified. Available updates are installed as soon as the update policy permits. A custom schedule and a maintenance window can be set (see further). If an update fails, the next attempt is made after at least 4 hours.

  • If a node reboots, crashes or hangs while a ReadyKernel patch is being installed, the patch is added to a list of banned patches. If the same happens while RPM packages are being installed, all packages from the RPM transaction are added to a list of banned packages. Banned patches or packages will not be installed in the future unless manually removed from the list.

  • Nodes are updated one by one. The node that has not been updated the longest is the first one in the queue.

  • Virtuozzo Storage nodes are automatically marked as suspended and not updated.

7.4.1. Enabling Smart Updates

To set up smart updates with a highly available lightweight Kubernetes cluster, do the following:

  1. Install the required RPM package:

    # yum install smart-updates
    

    This will install lightweight Kubernetes (k3s) among other dependencies.

  2. Set up a highly available Kubernetes cluster.

    Fault tolerance is ensured by keepalived. It provides a virtual IP address that is assigned to a control plane node. If it fails, the IP address migrates to a healthy control plane node.

    1. On each control plane node, prepare a service configuration file. Use a sample supplied by Kubernetes:

      # cp /usr/share/k3s/k3s_keepalived.conf.sample /etc/keepalived/keepalived.conf
      

      Replace the placeholders in keepalived.conf as follows:

      • ${INTERFACE} with the name of the node’s network interface that will negotiate for the virtual IP address. E.g., eth0.

      • ${AUTH_PASS} with a password, e.g., mypasswd. It must be the same on all control plane nodes.

      • ${APISERVER_VIP} with the virtual IP address at which the control plane will be available. E.g., 192.168.1.2.

    2. On each control plane node, prepare a health check script. It will be called periodically to check that the node holding the virtual IP address is operational. Use a sample supplied by Kubernetes as well:

      # cp /usr/share/k3s/check_apiserver.sh.sample /etc/keepalived/check_apiserver.sh
      

      Replace the placeholder in check_apiserver.sh as follows:

      • ${APISERVER_VIP} with the virtual IP address at which the control plane will be available. E.g., 192.168.1.2.

    3. On each control plane node, configure the firewall:

      # firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'
      # firewall-cmd --permanent \
      --add-port=8472/udp \
      --add-port=9443/tcp \
      --add-port=2379-2380/tcp
      # firewall-cmd --reload
      
    4. On the first control plane node, create the Kubernetes server configuration file and enable the keepalived, docker, and k3s-server services:

      # echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-server.env
      # echo -e "K3S_EXTRA_FLAGS=\"--cluster-init\"" >> /etc/k3s/k3s-server.env
      # systemctl enable keepalived docker k3s-server --now
      

      Where <my_secret_token> is an arbitrary secret string.

    5. On the second and third control plane nodes, create the Kubernetes server configuration files and enable the keepalived, docker, and k3s-server services:

      # echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-server.env
      # echo "K3S_URL=https://<virt_IP>:9443" >> /etc/k3s/k3s-server.env
      # systemctl enable keepalived docker k3s-server --now
      

      Where:

      • <my_secret_token> is the secret string from the previous step.

      • <virt_IP> is the virtual IP address from /etc/keepalived/keepalived.conf.

    6. On each agent node, open the required port, create the Kubernetes agent configuration file, and enable the docker and k3s-agent services:

      # firewall-cmd --permanent --add-port=8472/udp && firewall-cmd --reload
      # echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-agent.env
      # echo "K3S_URL=https://<virt_IP>:9443" >> /etc/k3s/k3s-agent.env
      # systemctl enable docker k3s-agent --now
      

      Where:

      • <my_secret_token> is the secret string from the previous step.

      • <virt_IP> is the virtual IP address from /etc/keepalived/keepalived.conf.

    7. Check that the cluster is healthy. For example:

      # kubectl get nodes
      NAME                  STATUS   ROLES         AGE     VERSION
      node1.example.local   Ready    etcd,master   3h36m   v1.19.7-k3s1
      node2.example.local   Ready    etcd,master   3h33m   v1.19.7-k3s1
      node3.example.local   Ready    etcd,master   3h33m   v1.19.7-k3s1
      
  3. Deploy the maintenance operator in the Kubernetes cluster:

    # kubectl apply -f /usr/share/smart-updates/maintenance-operator.yaml
    
  4. Wait until the maintenance operator is ready:

    # kubectl wait --namespace=vz-maintenance-operator \
    --for=condition=Complete job vz-maintenance-deployment-job-v<version> --timeout=1h
    job.batch/vz-maintenance-deployment-job-v<version> condition met
    

    Where <version> is the version of the smart-updates package, e.g., 1.0.0. You can also obtain the full job name from the maintenance operator configuration file. For example:

    # grep "vz-maintenance-deployment-job-v" /usr/share/smart-updates/maintenance-operator.yaml
     name: vz-maintenance-deployment-job-v1.0.0
    
  5. Create an UpdateManager instance to initiate synchronization of updates:

    # kubectl apply -f /usr/share/smart-updates/update-manager.yaml
    

7.4.2. Checking Node Status

You can now check the state of updates on nodes in the Kubernetes cluster with

# kubectl -n vz-maintenance-operator get updatemanager -o yaml

Or

# kubectl -n vz-maintenance-operator get updatemanager -o json

A typical response (abridged) may look like this:

<...>
"status": {
  "nodes": [
    {
      "id": "node1.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "status": "UpToDate"
      },
      "stageName": "stable"
    },
    {
      "id": "node2.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "numberOfPackagesToUpdate": 1,
        "status": "UpToDate"
      },
      "stageName": "fast"
    },
    {
      "id": "node3.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "status": "UpToDate"
      },
      "stageName": "slow"
    }
  ]
}
<...>

The sample above shows that all nodes are up-to-date, as per their update policies. It also reports that one RPM package has recently been installed on the second node, node2.example.local.

7.4.3. Setting the Maintenance Window and Schedule

With smart updates enabled, you can set one or more maintenance windows during which updating is allowed to start. The format is 0h00m, e.g., 1h30m for a window that is one hour and 30 minutes long.

You can also set an update schedule similar to that used by automatic updates. The format is standard Unix cron format, e.g., 0 1 * * 1-5 means “each day at 01:00 from Monday to Friday”.

Do the following:

  1. Open the update manager specification for editing:

    # kubectl -n vz-maintenance-operator edit updatemanagers default
    
  2. Add the required parameters to the spec part. For example:

    spec:
      <...>
      maintenanceWindows:
      - duration: 1h30m
        schedule: 0 1 * * 1-5
      <...>
    
  3. Save and close the specification.

7.4.4. Excluding RPM Packages and ReadyKernel Patches from Updates

You can prevent specific RPM packages and ReadyKernel patches from being installed during updating.

The format for RPM packages is <name>-<version>-<release>.<arch>, e.g., bash-4.2.46-34.vl7.x86_64. The format for ReadyKernel patches is X-Y-rZ, e.g., 123-4-r5.

Do the following:

  1. Open the update manager specification for editing:

    # kubectl -n vz-maintenance-operator edit updatemanagers default
    
  2. List the RPM packages and ReadyKernel patches in the spec section. For example:

    spec:
      <...>
      bannedPackages:
      - bash-4.2.46-34.vl7.x86_64
      bannedPatches:
      - 123-4-r5
      <...>
    
  3. Save and close the specification.

7.4.5. Changing Node Update Policies

To change a node’s update policy, re-label the node in the update manager specification as follows:

# kubectl label node <node> maintenance.virtuozzo.io/auto-updates-policy=<policy> --overwrite

For example:

# kubectl label node node1.example.local maintenance.virtuozzo.io/auto-updates-policy=fast --overwrite

Such re-labelling does not overwrite node update policies on the Virtuozzo update server. If smart updates are disabled, automatic updates will continue working as before.

7.4.6. Suspending Updates for All and Specific Nodes

You can prevent updates from being installed either across the entire Kubernetes cluster or on particular nodes. It may be useful to prepare the nodes for maintenance, for example.

Note

Virtuozzo Storage nodes get suspended from updating automatically. Updates will not be installed on Virtuozzo Storage nodes.

To suspend updating cluster-wide, set suspend: true in the update manager specification:

  1. Open the update manager specification for editing:

    # kubectl -n vz-maintenance-operator edit updatemanagers default
    
  2. Add the required parameters to the spec part. For example:

    spec:
      <...>
      suspend: true
      <...>
    
  3. Save and close the specification.

To suspend a specific node, assign the suspend label to it:

# kubectl label nodes node1.example.local maintenance.virtuozzo.io/suspend=""

The change will be reflected in the node status. For example:

# kubectl -n vz-maintenance-operator get updatemanager -o json
   "status": {
     "nodes": [
       {
         "id": "node1.example.local",
         "kernel": "3.10.0-1160.21.1.vz7.174.10",
         "readyKernel": {
           "reason": "ManuallySuspended",
           "status": "Suspended"
         },
         "rpmPackages": {
           "reason": "ManuallySuspended",
           "status": "Suspended"
         },
         "stageName": "stable"
       },
     <...>

To resume updates for a suspended node, remove the suspend label. For example:

# kubectl label nodes node.example.local maintenance.virtuozzo.io/suspend-

Again, the change will be reflected in the node status. For example:

# kubectl -n vz-maintenance-operator get updatemanager -o json
   "status": {
     "nodes": [
       {
         "id": "node1.example.local",
         "kernel": "3.10.0-1160.21.1.vz7.174.10",
         "readyKernel": {
           "status": "UpToDate"
         },
         "rpmPackages": {
           "status": "UpToDate"
         },
         "stageName": "stable"
       },
     <...>

7.4.7. Disabling Smart Updates

To disable smart updates and revert to automatic updates, do the following:

  1. Delete the Kubernetes entities related to the maintenance operator:

    # kubectl delete -f /usr/share/smart-updates/maintenance-operator.yaml
    
  2. If you deployed Kubernetes solely for smart updates and no longer need it, disable its server and agent services:

    # systemctl disable --now k3s-*
    

    Run this command on each node.