7.4. Updating Automatically with Smart Updates¶

7.4.1. Enabling Smart Updates¶

To set up smart updates with a highly available lightweight Kubernetes cluster, do the following:

1. Install the required RPM package:

   # yum install smart-updates
   

   This will install lightweight Kubernetes (k3s) among other dependencies.

2. Set up a highly available Kubernetes cluster.

   Fault tolerance is ensured by keepalived. It provides a virtual IP address that is assigned to a control plane node. If it fails, the IP address migrates to a healthy control plane node.

   1. On each control plane node, prepare a service configuration file. Use a sample supplied by Kubernetes:

      # cp /usr/share/k3s/k3s_keepalived.conf.sample /etc/keepalived/keepalived.conf
      

      Replace the placeholders in keepalived.conf as follows:

      • ${INTERFACE} with the name of the node’s network interface that will negotiate for the virtual IP address. E.g., eth0.

      • ${AUTH_PASS} with a password, e.g., mypasswd. It must be the same on all control plane nodes.

      • ${APISERVER_VIP} with the virtual IP address at which the control plane will be available. E.g., 192.168.1.2.

   2. On each control plane node, prepare a health check script. It will be called periodically to check that the node holding the virtual IP address is operational. Use a sample supplied by Kubernetes as well:

      # cp /usr/share/k3s/check_apiserver.sh.sample /etc/keepalived/check_apiserver.sh
      

      Replace the placeholder in check_apiserver.sh as follows:

      • ${APISERVER_VIP} with the virtual IP address at which the control plane will be available. E.g., 192.168.1.2.

   3. On each control plane node, configure the firewall:

      # firewall-cmd --permanent --add-rich-rule='rule protocol value="vrrp" accept'
      # firewall-cmd --permanent \
      --add-port=8472/udp \
      --add-port=9443/tcp \
      --add-port=2379-2380/tcp
      # firewall-cmd --reload
      

   4. On the first control plane node, create the Kubernetes server configuration file and enable the keepalived, docker, and k3s-server services:

      # echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-server.env
      # echo -e "K3S_EXTRA_FLAGS=\"--cluster-init\"" >> /etc/k3s/k3s-server.env
      # systemctl enable keepalived docker k3s-server --now
      

      Where <my_secret_token> is an arbitrary secret string.

   5. On the second and third control plane nodes, create the Kubernetes server configuration files and enable the keepalived, docker, and k3s-server services:

      # echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-server.env
      # echo "K3S_URL=https://<virt_IP>:9443" >> /etc/k3s/k3s-server.env
      # systemctl enable keepalived docker k3s-server --now
      

      Where:

      • <my_secret_token> is the secret string from the previous step.

      • <virt_IP> is the virtual IP address from /etc/keepalived/keepalived.conf.

   6. On each agent node, open the required port, create the Kubernetes agent configuration file, and enable the docker and k3s-agent services:

      # firewall-cmd --permanent --add-port=8472/udp && firewall-cmd --reload
      # echo "K3S_TOKEN=<my_secret_token>" >> /etc/k3s/k3s-agent.env
      # echo "K3S_URL=https://<virt_IP>:9443" >> /etc/k3s/k3s-agent.env
      # systemctl enable docker k3s-agent --now
      

      Where:

      • <my_secret_token> is the secret string from the previous step.

      • <virt_IP> is the virtual IP address from /etc/keepalived/keepalived.conf.

      Note

      You can repeat this step later to add new nodes to the Kubernetes cluster.

   7. Check that the cluster is healthy. For example:

      # kubectl get nodes
      NAME                  STATUS   ROLES         AGE     VERSION
      node1.example.local   Ready    etcd,master   3h36m   v1.19.7-k3s1
      node2.example.local   Ready    etcd,master   3h33m   v1.19.7-k3s1
      node3.example.local   Ready    etcd,master   3h33m   v1.19.7-k3s1
      

3. Deploy the maintenance operator in the Kubernetes cluster:

   # kubectl apply -f /usr/share/smart-updates/maintenance-operator.yaml
   

4. Wait until the maintenance operator is ready:

   # kubectl wait --namespace=vz-maintenance-operator \
   --for=condition=Complete job vz-maintenance-deployment-job-v<version> --timeout=1h
   
   job.batch/vz-maintenance-deployment-job-v<version> condition met
   

   Where <version> is the version of the smart-updates package, e.g., 1.0.0. You can also obtain the full job name from the maintenance operator configuration file. For example:

   # grep "vz-maintenance-deployment-job-v" /usr/share/smart-updates/maintenance-operator.yaml
   
   name: vz-maintenance-deployment-job-v1.0.0
   

5. Create an UpdateManager instance to initiate synchronization of updates:

   # kubectl apply -f /usr/share/smart-updates/update-manager.yaml
   

7.4.2. Checking Node Status¶

You can now check the state of updates on nodes in the Kubernetes cluster with

# kubectl -n vz-maintenance-operator get updatemanager -o <yaml|json>

A typical response (abridged) may look like this:

<...>
"status": {
  "nodes": [
    {
      "id": "node1.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "status": "UpToDate"
      },
      "stageName": "stable"
    },
    {
      "id": "node2.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "numberOfPackagesToUpdate": 1,
        "status": "UpToDate"
      },
      "stageName": "fast"
    },
    {
      "id": "node3.example.local",
      "kernel": "3.10.0-1160.21.1.vz7.174.10",
      "readyKernel": {
        "status": "UpToDate"
      },
      "rpmPackages": {
        "status": "UpToDate"
      },
      "stageName": "slow"
    }
  ]
}
<...>

The sample above shows that all nodes are up-to-date, as per their update policies. It also reports that one RPM package has recently been installed on the second node, node2.example.local.

7.4.3. Setting the Maintenance Window and Schedule¶

You can set one or more maintenance windows during which updating is allowed to start. The format is 0h00m, e.g., 1h30m for a window that is one hour and 30 minutes long.

You can also set an update schedule similar to that used by automatic updates. The format is standard Unix cron format, e.g., 0 1 * * 1-5 means “each day at 01:00 from Monday to Friday”.

Do the following:

1. Open the update manager specification for editing:

   # kubectl -n vz-maintenance-operator edit updatemanagers default
   

2. Add the required parameters to the spec part. For example:

   spec:
     <...>
     maintenanceWindows:
     - duration: 1h30m
       schedule: 0 1 * * 1-5
     <...>
   

3. Save and close the specification.

7.4.4. Setting the Repositories¶

RPM package updates are obtained from default repositories listed early in this section. You can, however, change them separately for each policy. If you do so, only the repositories that you provide will be used by smart updates. You will still be able to manually install RPM packages from unlisted repositories.

Do the following:

1. Open the update manager specification for editing:

   # kubectl -n vz-maintenance-operator edit updatemanagers default
   

2. Add the needed repository IDs under the repositories key for the desired policy. Add the key if needed.

   Note

   You can get repository IDs from yum repolist.

   For example, to use only VzLinux repositories for the fast policy:

   spec:
     stages:
     - name: fast
       nodeSelector:
         matchLabels:
           maintenance.virtuozzo.io/auto-updates-policy: fast
       repositories:
       - virtuozzolinux-base
       - virtuozzolinux-updates
   

3. Save and close the specification.

7.4.5. Excluding RPM Packages and ReadyKernel Patches from Updates¶

You can prevent specific RPM packages and ReadyKernel patches from being installed during updating.

The format for RPM packages is <name>-<version>-<release>.<arch>, e.g., bash-4.2.46-34.vl7.x86_64. The format for ReadyKernel patches is X-Y-rZ, e.g., 123-4-r5.

Do the following:

1. Open the update manager specification for editing:

   # kubectl -n vz-maintenance-operator edit updatemanagers default
   

2. List the RPM packages and ReadyKernel patches in the spec section, under bannedPackages and bannedPatches keys, respectively. For example:

   spec:
     <...>
     bannedPackages:
     - bash-4.2.46-34.vl7.x86_64
     bannedPatches:
     - 123-4-r5
     <...>
   

3. Save and close the specification.

To allow installation of banned patches and packages again, remove them from the list.

7.4.6. Changing Node Update Policies¶

To change a node’s update policy, re-label the node in the update manager specification as follows:

# kubectl label node <node> maintenance.virtuozzo.io/auto-updates-policy=<policy> --overwrite

For example:

# kubectl label node node1.example.local maintenance.virtuozzo.io/auto-updates-policy=fast --overwrite

Such re-labelling does not overwrite node update policies on the Virtuozzo update server. If smart updates are disabled, automatic updates will continue working as before.

7.4.7. Suspending Updates for All or Specific Nodes¶

You can prevent updates from being installed either across the entire Kubernetes cluster or on particular nodes. It may be useful to prepare the nodes for maintenance, for example.

To suspend updating cluster-wide, set suspend: true in the update manager specification:

1. Open the update manager specification for editing:

   # kubectl -n vz-maintenance-operator edit updatemanagers default
   

2. Add the required parameters to the spec part. For example:

   spec:
     <...>
     suspend: true
     <...>
   

3. Save and close the specification.

To suspend updates on a specific node, assign the suspend label to it:

# kubectl label nodes node1.example.local maintenance.virtuozzo.io/suspend=""

The change will be reflected in the node status. For example:

# kubectl -n vz-maintenance-operator get updatemanager -o json
   "status": {
     "nodes": [
       {
         "id": "node1.example.local",
         "kernel": "3.10.0-1160.21.1.vz7.174.10",
         "readyKernel": {
           "reason": "ManuallySuspended",
           "status": "Suspended"
         },
         "rpmPackages": {
           "reason": "ManuallySuspended",
           "status": "Suspended"
         },
         "stageName": "stable"
       },
     <...>

To resume updates for a suspended node, remove the suspend label. For example:

# kubectl label nodes node.example.local maintenance.virtuozzo.io/suspend-

Again, the change will be reflected in the node status. For example:

# kubectl -n vz-maintenance-operator get updatemanager -o json
   "status": {
     "nodes": [
       {
         "id": "node1.example.local",
         "kernel": "3.10.0-1160.21.1.vz7.174.10",
         "readyKernel": {
           "status": "UpToDate"
         },
         "rpmPackages": {
           "status": "UpToDate"
         },
         "stageName": "stable"
       },
     <...>

7.4.8. Disabling Smart Updates¶

To disable smart updates and revert to automatic updates, do the following:

1. Delete the Kubernetes entities related to the maintenance operator:

   # kubectl delete -f /usr/share/smart-updates/maintenance-operator.yaml
   

2. If you deployed Kubernetes solely for smart updates and no longer need it, disable its server and agent services:

   # systemctl disable --now k3s-*
   

   Run this command on each node.