10.7. Setting Up Docker and Kubernetes in Virtuozzo Containers

You can run Docker and Kubernetes inside Virtuozzo Hybrid Server system containers.

In the current version of Virtuozzo Hybrid Server, you can use system containers based on CentOS 7 as well as Ubuntu 18.04 and 20.04.

Note

Modules and third party add-ons that depend on operations prohibited in containers (loading of kernel modules, mounting of block devices, direct access to physical hardware) may not work in containers.

Essentially, you need to:

  1. Enable and start the vz-k8s-inside-ct service that configures the hardware node for running Kubernetes and Docker swarm mode in system containers.

  2. Create a system container with a network connection and enough resources to run the Docker containers you need.

  3. Install Docker inside the system container according to the official documentation. It is recommended to change the cgroup driver to systemd.

  4. Install Kubernetes inside the system container according to the official documentation.

  5. Create more similar system containers and join them into a Kubernetes cluster or a Docker swarm.

These steps are demonstrated in more detail in the following tutorial:

  1. Enable and start the vz-k8s-inside-ct service that configures the hardware node for Kubernetes and Docker swarm mode:

    # systemctl enable vz-k8s-inside-ct && systemctl start vz-k8s-inside-ct
    

    Note

    Restart running containers to apply these configuration changes to them.

  2. Create a system container with a supported guest OS. For example:

    # prlctl create dockerct1 --vmtype ct --ostemplate centos-7-x86_64
    
  3. Provide the system container with enough resources. For example, add more RAM:

    # prlctl set dockerct1 --memsize=2G
    Set the memsize parameter to 2048Mb.
    
  4. Disable swap in the system container as required for Kubernetes. For example:

    # prlctl set dockerct1 --swap=0
    Set swappages 0
    
  5. Set up network in the system container. For example, add a network adapter connected to a bridged network (by default) and enable DHCP for it:

    # prlctl set dockerct1 --device-add net --dhcp yes
    Enable automatic reconfiguration for this network adapter.
    Creating net0 (+) dev='' ifname='eth0' network='Bridged' mac=<...> card=virtio dhcp='yes'
    Created net0 (+) dev='veth72d71c9b' ifname='eth0' network='Bridged' mac=<...> dhcp='yes'
    
  6. Set a unique hostname for the system container. For example:

    # prlctl set dockerct1 --hostname dockerct1.example.local
    

    Note

    Make sure to register the hostname with the DNS server.

  7. Set the root password for the system container to be able to log in via SSH:

    # prlctl set dockerct1 --userpasswd root:<passwd>
    

    Where <passwd> is a strong password.

  8. Start and log in to the system container:

    # prlctl start dockerct1
    # ssh <dockerct1_IP>
    

    Where <dockerct1_IP> is the system container’s IP address. If you previously enabled DHCP, you may need to find out the IP address. For example:

    # prlctl exec dockerct1 ip -4 a | grep inet
    <...>
    inet 10.194.110.194/16 brd 10.194.255.255 scope global dynamic eth0
    

    Perform the next steps inside the system container.

  9. Open the ports required by Kubernetes and the Docker swarm:

    • For Kubernetes control-plane:

      # firewall-cmd --permanent --add-port=6443/tcp;\
      firewall-cmd --permanent --add-port=2379-2380/tcp;\
      firewall-cmd --permanent --add-port=10250/tcp;\
      firewall-cmd --permanent --add-port=10251/tcp;\
      firewall-cmd --permanent --add-port=10252/tcp;\
      firewall-cmd --reload
      
    • For the Docker swarm:

      # firewall-cmd --permanent --add-port=2377/tcp;\
      firewall-cmd --permanent --add-port=7946/tcp;\
      firewall-cmd --permanent --add-port=7946/udp;\
      firewall-cmd --permanent --add-port=4789/udp;\
      firewall-cmd --reload
      
  10. Install Docker as laid out in the official guide. Before you start the Docker service, however, change the cgroup driver to systemd:

    # mkdir /etc/docker
    # cat > /etc/docker/daemon.json <<EOF
    {
      "exec-opts": [
        "native.cgroupdriver=systemd"
      ],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2",
      "storage-opts": [
        "overlay2.override_kernel_check=true"
      ]
    }
    EOF
    
  11. Enable and start the Docker service:

    # systemctl enable docker && systemctl daemon-reload && systemctl start docker
    

    Note

    If the Docker service fails to start with the non-mandatory option storage-opts, remove this option. The final configuration is as follows:

    # cat > /etc/docker/daemon.json <<EOF
    {
      "exec-opts": [
        "native.cgroupdriver=systemd"
      ],
      "log-driver": "json-file",
      "log-opts": {
        "max-size": "100m"
      },
      "storage-driver": "overlay2"
    }
    EOF
    
  12. Verify that Docker Engine works:

    # docker run hello-world
    Unable to find image 'hello-world:latest' locally
    latest: Pulling from library/hello-world
    0e03bdcc26d7: Pull complete
    Digest: sha256:4cf9c47f86df71d48364001ede3a4fcd85ae80ce02ebad74156906caff5378bc
    Status: Downloaded newer image for hello-world:latest
    
    Hello from Docker!
    This message shows that your installation appears to be working correctly.
    <...>
    
  13. Install kubelet, kubeadm, and kubectl packages according to the official documentation.

  14. Initialize the Kubernetes control-plane node. For example:

    # kubeadm init
    <...>
    Your Kubernetes control-plane has initialized successfully!
    
    To start using your cluster, you need to run the following as a regular user:
    
      mkdir -p $HOME/.kube
      sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
      sudo chown $(id -u):$(id -g) $HOME/.kube/config
    
    You should now deploy a pod network to the cluster.
    Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
      https://kubernetes.io/docs/concepts/cluster-administration/addons/
    
    Then you can join any number of worker nodes by running the following on each as root:
    
    kubeadm join 10.194.110.194:6443 --token qrdi2v.t4rr9yyylpgpnkcw \
        --discovery-token-ca-cert-hash sha256:a7d0e65<...>
    
  15. As you are the root user, run

    # export KUBECONFIG=/etc/kubernetes/admin.conf
    
  16. Deploy a pod network. For example:

    # kubectl apply -f "https://cloud.weave.works/k8s/net?\
    k8s-version=$(kubectl version | base64 | tr -d '\n')"
    serviceaccount/weave-net created
    clusterrole.rbac.authorization.k8s.io/weave-net created
    clusterrolebinding.rbac.authorization.k8s.io/weave-net created
    role.rbac.authorization.k8s.io/weave-net created
    rolebinding.rbac.authorization.k8s.io/weave-net created
    daemonset.apps/weave-net created
    
  17. Check that Kubernetes has been set up correctly. For example:

    # kubectl get pods --all-namespaces
    NAMESPACE     NAME                                             READY  STATUS   RESTARTS  AGE
    kube-system   coredns-f9fd979d6-29k27                          1/1    Running  0         5m
    kube-system   coredns-f9fd979d6-f6xd5                          1/1    Running  0         5m
    kube-system   etcd-dockerct1.example.local                     1/1    Running  0         5m
    kube-system   kube-apiserver-dockerct1.example.local           1/1    Running  0         5m
    kube-system   kube-controller-manager-dockerct1.example.local  1/1    Running  0         5m
    kube-system   kube-proxy-559tc                                 1/1    Running  0         5m
    kube-system   kube-scheduler-dockerct1.example.local           1/1    Running  0         5m
    kube-system   weave-net-jzm8p                                  2/2    Running  0         5m
    
  18. Create a Docker swarm using the system container’s IP address. For example:

    # docker swarm init --advertise-addr 10.194.110.194
    Swarm initialized: current node (n1slc7on484f7ak4d3vaz6tk4) is now a manager.
    To add a worker to this swarm, run the following command:
        docker swarm join --token SWMTKN-1-<...> 10.194.110.194:2377
    To add a manager to this swarm, run 'docker swarm join-token manager'
    and follow the instructions.
    
  19. Create more (e.g., two more) system containers with the same configuration as explained in steps 2 to 8. They will run Kubernetes and Docker swarm workers.

    Perform the next steps in each of the new system containers.

  20. Open the ports required by Kubernetes and the Docker swarm:

    • For the Kubernetes worker:

      # firewall-cmd --permanent --add-port=10250/tcp;\
      firewall-cmd --permanent --add-port=30000-32767/tcp;\
      firewall-cmd --reload
      
    • For the Docker swarm:

      # firewall-cmd --permanent --add-port=2377/tcp;\
      firewall-cmd --permanent --add-port=7946/tcp;\
      firewall-cmd --permanent --add-port=7946/udp;\
      firewall-cmd --permanent --add-port=4789/udp;\
      firewall-cmd --reload
      
  21. Install Docker and Kubernetes as explained in steps 10 to 13.

  22. Add workers to the Kubernetes cluster with the command reported by kubeadm init. For example:

    # kubeadm join 10.194.110.194:6443 --token qrdi2v.t4rr9yyylpgpnkcw \
          --discovery-token-ca-cert-hash sha256:a7d0e65<...>
    
  23. Add workers to the Docker swarm with the command reported by docker swarm init. For example:

    # docker swarm join --token SWMTKN-1-<...> 10.194.110.194:2377
    This node joined a swarm as a worker.
    
  24. Log in to the system container running the control plane and swarm manager (dockerct1 in this tutorial) and check that both the Kubernetes cluster and Docker swarm work. For example:

    # kubectl get nodes
    NAME                      STATUS   ROLES    AGE   VERSION
    dockerct1.example.local   Ready    master   10m   v1.19.2
    dockerct2.example.local   Ready    <none>   5m    v1.19.2
    dockerct3.example.local   Ready    <none>   5m    v1.19.2
    
    # docker info
    <...>
    Swarm: active
     NodeID: n1slc7on484f7ak4d3vaz6tk4
     Is Manager: true
     ClusterID: lyz9u13zoswtm03jkswqcdzaj
     Managers: 1
     Nodes: 3
     Default Address Pool: 10.0.0.0/8
     SubnetSize: 24
     Data Path Port: 4789
     Orchestration:
      Task History Retention Limit: 5
     Raft:
      Snapshot Interval: 10000
      Number of Old Snapshots to Retain: 0
      Heartbeat Tick: 1
      Election Tick: 10
     Dispatcher:
      Heartbeat Period: 5 seconds
     CA Configuration:
      Expiry Duration: 3 months
      Force Rotate: 0
     Autolock Managers: false
     Root Rotation In Progress: false
     Node Address: 10.194.110.194
     Manager Addresses:
      10.194.110.194:2377
    <...>
    

After completing this tutorial, you have three or more Virtuozzo Hybrid Server 7 system containers running Docker and Kubernetes. You have also created a Kubernetes cluster and a Docker swarm.